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I. INT RODUCTION 

Argument's about the appropriate federal role in U.S. education are as 
old as public educ^on itself, but the arguments became more intense as 
'federal involvement in education grew. In the late 1950s and early 1960s 
these x arguments centered on the political wisdom of federal intervention. 
In the late 1960s and 1970s, as evidence on the effects pf the new Great 
Society education programs began to accumulate, the debate seemed to center 
more on whether federal intervention did any good, whatever one thought" 
about its vvisdom. But throughout these years, the federal role in education 
expanded, under both Democratic ajid Reprhlican administrations. The 1980 
presidential campaign once again placed the issue of political principle in 
the forefront, and since then the Reagan administration has begun try 

ing Jo^ismantle the results of two decades of growth, on the grourtds that 
education should be chiefly a state and local responsibility. 
* Many issues intersect in these controversies, and disentangling them 
all would be no small task. One issue that has surfaced from time Xo^time 
concerns the competenq/e of the central government: what can , it do we^l, 05 
at least better than anyone else? Competence is of course not the only con- 
sideration in allocating political responsibilities, but it is not a trivial 
consideration. Competence can change with experience or ^i4sources, but there 
often are structural constraints on the developmei^ of competence, or great 

> • . ' / ' - - 

differences in the costs of developing it as one, level of government rather 
than another. In this essay, we explore the c$npetence of federal agencies 
in education^ by exploring federal-local interaction in one education program 

One advantage of focusing on the competence of government agenfcies is 
that it 'directs, attention to what federal government has done well, and 



poorly, in practice, 'Liberals havfe tended to assume that the government 
could dd a good job of whatever it tried in education, if only the resources 
were available, while conservatives assume' that a good job would be unlikely. 
Indeed, they believe that even if~the central government could do a good job 
it woul^ fre a^ad^tlung, because of the effects on the distribution of power 
within th£ federal system. Most of these arguments Have occurfed at the 
rarefied level of political principle. • While we care ^bout principle, the ■> 
question of competence can best be *il luqunated by booking at practice. We 
consider the actual activities of federal and -local educators, and the char- 
acter of educational practice at the federal and local levels." Our story 
centers on the interactions among practitioners who work in these two very 



different parts of American education. 



The case in point---the ' Experimental Schools Program--was modest 
in size, but broad in scope and ambition. ESP had a small total appropriation 

of £55 Trillion dollars--and -supported projects in only 18 communities. 

But it was a highly touted effort by the U.S. Office of ^Education to bring 

"comprehensive" reform to local schools, and «to create a new model of federal 

relations with local schools. ESP grew out of discontent with {fre^t Soc?.e£y 

education *pol icy and practice, and represented an effort to marry, the sweep- 

ing visions of change 'associated with th^Jehnsonian programs to the concern 

for local ^ontrol associated with more conservative views. I The program 

* » * 

began in 1971, "but terminated in gr^at disappointment ,and disarray a mere 

five years later. " * ■ m ^ 

'Scrutiny of 4he< Experimental \ Schools ' ' Program ' will 
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illuminate issues., of federal - competence, but it will not 
offer x ja conclusive answer /to questions about the federal 
role. There is no sifch thing. Our account will throw only a bit of light 
on fe complicated and continuing problem. We begin with a brief analysis of 
federal -local relations in education. : 

// 

Federal - Local Re 1 at ions • \ 

The federal role in education is ambiguous: since the late 1950s federal 
agencies have been charged increasingly with reforming education at the local 
level, but'their resources for -influencing local education are limited. The 
imbalance hetueen federal mandates and federal Resources has created many 
curious twists *in federal - local relations. From a fiscal perspective, the 
relation seems plain enough: local schools receive a modest portion of their 
operating revenues from federal agencies, and federal education agenciM^ Jjfl- 
pose various requirements for? rjsform . in return fpr the mpney. Typically, we 
think the local agencies are dependent upoh their federal counterpart: the 
locals, after all, need .the* money. That is true enough, but th^re is a 
mutual obligation that implies a further dependence, one often unnoticed: * 
for the federal agency's work to succeed, local schools must comply with 
federal requirements. If local school authorities depend on federal agen- 
cies for money, the federal agencies depend on local Authorities to accom- 
plish federal purposes. 

/ The character of this dependence is complicated by several deep dif- 
ferences between federal and local educational practice. For one thing, 
he units of practice are different. Federal educatioj managers have a 
national obligation, and ^their unit bi practice is the federal program.^ 
They seek success for these programs, which means that to-some extent they 



must try to engage local districts* in operations consistent with* the program 
goals and values. They wish local districts to coordinate internal efforts 
toward those' goals. The .local units of. practice, however, are schools and 
classrooms. There are tens of thousands- of these, in the United States-- 

indeed, in any single large district there are thousands of classrooms. The 

» V. 

numbers are so ln^e] and the aims of education and the nature of teaching 

at ■ . * 

are so indeterminate, that even central administrators at the local level 
•must leave teachers arid principals 'a great deal of autonomy to shape their 
/ovnti practice. Education is coordinated at the local level, but in many cru- 
cial respects itJLs only weekly coordinated, at ^ least by means of thie— formal 
local governmental and administrative apparatus. Coordination ( of local units 
o'f practice in federal program^' is thus problematic from the outset. ' 

In addition to differences in the units of practice, federal and local 
educators also differ concerning the aims of practice. Since the late 1950s, , 
federal education managers have fcjeen preoccupied with the reform of local S 
scho^Ts. Several factors contributed to the widespread assumption that 

}ocal school reform is the central task of federal education agencies. 
One was external pressure on the federal government to promote change in 
local school policy and practice, because such c-hange had been effectively 
.blocked at. the state or local level. The desegregation of southern schools 
is the best example of this phenomenon. A related factor is that Washington 
has become the nation's political center. The increasingly national charac- 

ter of politics and culture, and the widespread sense that the national gov- 

{ 

etnmentis the most powerful lever for social change, have encouraged those 
with am interest in reform of all sorts — even conservative reform — to grav-i- 
tate tp* the national capital. And finally, there is the view, especially 



common In education, that local government is inadequate, perhaps even bank- 
rupt. These developments and others encouraged the belief in recent 

decades that the unique responsibility of federal policy in education is to 
correct the errors of local practice. \ , ) 

Refas^n does not figure so prominently in the work of local teachers' 

4 ' 

f * V * . » 

and administrators. While many 6eek change in one way or another, their 
practice centers much more on providing complex services in an established 
social institution. Quite naturally, they are t more preoccupied with- 
^hat sociologists call maintenance than with change, for they must attend 
to all the continuing work of schools: they -must select books-, grade papers, 
teach subjects, keep schools heated, and hire and fixe staff. ' Federal edu-* 
cation officials, by contrast, have no schools to run, nor any of th^ associ- 
ated^ tasks. They have federal programs to devise, administer, and change. 
Thus, not only do the units of practice differ greatly, but the nature of 
practitioners 1 work, and their responsibilities, differ as well. w Powerful 
forces act upon local practice to press it 4 away from an extensive focus on 

reform, while recent pressures on federal practice entourage precisely thi* 

*t 

focus. % • * 

Thus, the aims of federal practice have pressed feSeral agencies toward 

' .. . • N • 

the reform of local practice, tohil$ differences in federal and local prac- 
tice impede federal influence. These differences affect federal-local 
relations, but their effect varies according 'to-the nature of thfe pro'grams 
and policies over which relations occur. A federal - program wfrose aim is 
simply to give financial support for whatever local' education agencies wish 
to do--such a? the Impact Aid Program— depends only modestly on local agencies 



for success in its own -performance; the Impact Aid Program is successful if LEAs 

simply accept *the,money affd spend ^t on education,, But a federal pro&rarii that 

aims to retorm local practice by spending mone/ to improve students*)] school achi 

ment--such as Title I of the 1965 ESEA— depends much moie extensively upon 

1 > 
lo % cal practice. In order for such a federal ,progranv to succeed, not only 

must money be spent by local edudrtional authorities', but it must be spent 

on activities li\e reading, .and spent in such a way that students 1 reading* 

^improves. That, in turn, means that there must afso be changes in how 

teachers and other local practitioners do their work, and perhaps changes 

as well in the organization in which that work goes on. 

'In the case of the Title I program, then, the success of fedl^al man- 



-agers* practice depends on. the success of many local practitioners' work, 

f ■ ' - 

And the l<?cal work is complex,^ including not only teaching and learning 

reading, but managing those tasks, changing them, and quite likely also 

managing school organization. 'This is in marked contrast to unrestricted 

aid, where the success' of federal practice y^ould depend much less on the 

success of focal practice. The chief requirement in the latter case woulc^ 

* • / , 

be local acceptance of federal monies and their successful expenditure on 

education: 'whether the activities on which the funds are expended were suc- 
v * . . ? 

cessful or not is probably irrelevant to whether the federal program has , 
succeeded, or whether federal officials' work was competent. Since unre- 

stricted aid would by definition jiot be targeted, and would fye spent on 

* * ♦ * 

many diffe^mt loca^l activities' federal responsibility for local practice i 
would be gJE|ly attenuated, and quite diffuse. There might be a sense in 
which the value of such aid would seem to be related to lihe quality of lqcal^ 
educatiqn--i.e. , if local education fell into wide'disrepute, the aid might 



/ 



be questioned. But. in the case of programs like Title I, there would be a 
much greater and m6re specific federal depcn^nce upon local practice; the 
program aims imply that the federal agency wili improve the reading of local 
students, and the wcn^k of local practitioner^in teaching and organizing 
reading. In some sense, federal officials become responsible for the .work 
of local practitioners. Federal effectiveness mast include. the capacity to 
improve the % ef fectiveness of their local partners. That is a tall order. 

There is no reason in principle why such an order* could not be filled, 
if federal agencies have' the wherewithals-technical , political^ .or jftscal-- 
to produce success in local schools. But if federal, agencies lack the means 
to dd so, then they will be seeking the, reform of local practice without * 
being able to muster .the ^^ources Inquired to make the 'reforms a success. 

4 

Federal agencies would then depend on local practitioners to make many "com- 
plicated changes i-n their work/ without being able, to provide adequate means 
to. induce or support those changes. 

What' resources do federal education agencies have at their disposal, ' 
to reform local* practice? MQney is one: if federal agencies paid much of* 
the freight for local educati,on--if they could increase or cut local bud- 
gets by large ysinounts-- they mig^it then h^ve. considerable influence on local 
practice. Political ^authority i5 another resource, for authority, however^ 
derived, would encourage local agencies to conform to federal mandates. * 
A third resource is persuasion. ' Even if local schools have the 
dates. A third resource i$ persuasion. For even i£ lofcal schools have the 
other resources tfiey : need to carry out federal mandates, *they jnay lack the 
desire to do so.' Or, having the desire, they may lack the know-how. federal 
agencies might remedy these 'problems by persuasion* They could reason with 



local educators, convincing them by rational 'argument or appeals to morality • t 

£hat federal mandates should be carried out. Or federal officials could 

remedy the lack of local know-how by supplying it to the deficient locali- 

ties, or by providing inducements For the localities to acquire it them- * * 

» « 

1 ■ ■ 

selves, or both. 

Federal education agencies have used all of these resources in their 
efforts to secure local cjompliance.jwith federal policy ,^n educartion. One 
diff iculty in the Lr efforts , though,, has been the considerable limits on 
federal authority in education:- by law and tradition, most authority in 
education lies at the state and local level.* Authority is an extremely 
limited federal resource. Only in a few extraordinary situations has tfcr - ^ 
fe/deral government commanded the moral or political resources to y exercise 
authority effectively. Another difficulty has been ttiat federal monies, 
while much increased over earlier years, are s-tirl a modest fraction of 
local revenues. Local districts*. value federal funds because they offer 
room for maneuver at the margin, and this provides federal' education 

agenties with some leverage on localities. But the leverage has been^lim* 
ited,"" in part by the size of the federal contribution. Jn these two respects, 
then, federa v l resources for influencing local action have^ been only modest. 

As a result, federal education agenpies have been^ chronically over- 

i 

extended: the scope^ of their mandates for local change typically far exceeded the 

money and authority the central government could bring tp bear on state and 

6 • • 

local schooP'agencies, The only exception to this pattern, briefly, wa$ 

Title VI of the 1964 Ci^yil Rights Act. Fpr a few years during the middle and late t 
\ 

1960s, federal officials 'had something so powerful to offer local practi- f . 



tiqners—namely, the absence of fiscal, political, and legal trouble, 
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p. 



V 

perhaps even the absence of civir-strife--that # local officials were encour- 

aged, often against tfreir will, ^o comply with federal directives , As a 

result, many sout|wn schools were desegregated., 

But in most cases, federal officials have haci less to offer: statutory 

requirements for .participation; some ideas aboyt how to do it; regulations*; 

and a modest amount of money. These instruments arq sufficient to produce 

the local changes federal of ftc idls desired when local practitioners both / 

wished a particular change and had the capacity to produce it. This was 

the case in many instances ,. and loc^^practice changed. In Jjany Qther 

instances'* c however, the locals. did not N vrish the changes federal agencies 

desired, or*did not wish all the federally mandated changes, ^ or did not 

wish to- make the changes in the ways required! or" could not make the changes 

despite their wish to do so. In such cases; fecteral agencies depended on 
* ' . + 

local schools to reform local practice, when either the local will to make 
the changes, or the local ability to change, or both were in v question. 
Federal agencies were asserting national priorities^ but because they 
lacked the resources to insure local compliance wit,h these priorities, th£ 
success of federal programs same to depend * upon lbcal agencies whose ca- 
pacity for change wa»s doubt/ful. ThBt is a curious sort of dependence. 

One~result Qf this situation was a peculiar pattern in fede*ral-local 
political relations. Because local educators could see the 

imbalance between federal mandates and federal resources, they knew that 
in .fact they had considerable discretion in carrying out those mandates, 
even when the regulations, and guidelines might suggest otherwise. • From / 
this angle, *Title I of the 1965 ESEA, and PL 94-142, which were treated in 
regulations and guidelines as though they were highly restricted categorical 
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programs, 'might be thought of as beings moderately restricted block grants • * 
in practice. . * * * 

, Such incongruence is not necessarily a problem, if everyone 
understands the limits it implies, and observes a decent respecter the 
proprieties. But the federal preoccupation with reform often led to a curious 
interpretation of events: because many federal officials saw their task as 

4 

the improvement of local practice that was sadly deficient, they were inclined 
to interpret local behavior in terms of th'is presumed deficiency. 'Federal 
officials tended to explain poor local performance in federal programs 

i A 

as another* manifestation of local deficiency. Other expla- 
* 

nations, such as overambitious program aims, ^or insufficient program 'resources 
were frequently ignored. a result, in many cases when federal programs', 
appeared to produce" poor results, federal executives and many other observers 
concluded that the problem lay at the local level, p and that the solution was 
a stronger dose of the previous medicine. . This approach led to 
more ideas for *e£orm of local prap^4ee,. and to more statutes, regulations, 
and guidelines. The imbalance between federal requirements and federal* 
resources was not corrected, though, * so that as federal requirements for 
local reform .mounted, federal < resources remained only a modest 

• i 

contribution to the local costs of reform. The ensuing imbalance 

r * 

insured that many federal ambitions would 'fail of achievement, and would them- 
selves be seen as evidence^ of the need . for *even more ambitious federal action. 

* "*\ * 

The Experimental Schools Program it a particularly nice case of tttis crazy"' 

«* ■ 

logic, m in which federal reformers,, seemed incapable of ^imagining any other 
reason fot their ^program's problems than_ loca^incapacity. 

A second result of the dependence of federal reformers on local agencies "is 
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a need to*rely^$>ri cither than 'fiscal and political resources as a means of 

accomplishing federal objectives, namely on persuasion. Persua^oa has many 

forms—including everything from the use of rational argument to 'decide aboht a 

program of research, to brainwashing an entire people. In the case of federal 

• , I , ' \ 

education programs, the particular form of persuasion that federal authori- 

• *■ 

ties relied on was social science. . One hope behind the federal investment 

* •< * . » 

in social science was that*research could identify the solutions to educa- 
tional problems. Another hope was that if researchers could 'identify better 
teaching methods, or produce better /curricula, the knowledge itself would 
stimulate local action. Consequently, as federal officials pressed for local 
school reform they made extraordinary efforts to dev^p sqcial science kij^- 
edge aboutTwhat good educational practice is, and how it can be promoted. 
Since the .early 1960s, when federal education officials have .assessed the * 
schools 1 shortcomings they haye identified the absence of knowledge about 
good practice and how to produce it as a^ key problem. And repeated efforts 
.-have been m&de, in one program after another, to remedy this problpnTfcy more 
and Setter research and evaluation. This pattern has been evident* in nearly 
everything from early efforts to establish R6D Centers and Regional Edupa- 
tional Laboratories designed to develop netf knowledge^ that would improve local 
practice, to recent attempts to improve evaluation in such programs as Title I 
ESEA, by developing canned M models ,fV for evaluation and supporting large train- 
ing and technical assistance agencies to help local schools to use these models 
to iipprove Title I evaluation. Knowledge, after all, is a r^sou^ce that federal 
practitioners might, well use in large , amount s , even 'if money and' political 
power were in short supply, flhat federal education- agencies lacked in •fiscal 
and gplitical clout, they tried to make up wi£h scientific knowledge. 

■ i • " & 

• 0 .13 . 
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Federal officials turned /to applied social science on the assump- 

tion that it could be authoritative in the quest for improved local practice. 

. / • « 
In -contrast to the patchy, anecdotal, ofteji inarticulate and hand-me-down 

* 1 i 

knowledge of local- practitioners, social science was believed to be capable - 
of demonstrating, in a scientifically conclusive way, which local practices 
worked, and why. Because it would be conclusive, sacial science was expected, 
to provide compelling, inducements for action to improve schools. 
But as it turned out, social science was not conclusive about what programs 
worked or didn't: doing more evaluation did not iSecfease scientific di-sagreement 

about what worjcs, or what we mean by "works," or why some things work and 

*- § , 

others don't. If anything, evaluations and other studies increased controversy 

! 

on these points. Social science did not establish an authoritative basis for 
improving practice, and thus did not provide federal education agencies with 
a reliable resource for reforming local practice. 

Curiously enough, though, regularly poor results with evaluation did. 
not discourage federal reliance on social science as an instrument 
of. policy. Instead, the failures appear to have had the opposite " effect : * 
they were taken af a sign^that more applied research was needed, in order to 
produce the desired knowledge, and thereby the reforms of local practice. 

The history^of federal efforts to reform local practice was thus marked ■ 
by more and more disappointing results frwn social science, yet at the same 
time, by ever more projects for local, reform through social science, and more 
money with which to do the job. The respons^e to the failure of social science 
was demands for more and better social science. .One important consequence was> 
a rapid growth of federal research and development efforts into many corners 
of education. Another was an extraordinarily broad definition of 
the tasks of research and development. As. the poor results of some applied 



13 



research become apparent, federal administrators e>panded the tasks^of research 
even further, in a search for ways to compensate for the unexpected failures' 
The response, for example, to failures of jarly program evaluation p 

was a flurry of attempts to scientifically distinguish successful from unsuj- 

■ 4 ' ' I J 

cessful -local projects, to determine what produced success in the happy cases and 



how it* could be replicated. The ori^Uial scien 



ific &genda--determining 



whether a particular program succefeded--gave way, upon its failure, to an ex- 
panded agenda: finding projects -that work, figuring out why some projects suc : 
ceed and other fail, and figuring out how to get complex social institutions 
to. replicate success. There were many instances of :his phenomenon. The wish to 
understand' social problems is reasonable, ^ut it was perverted by a crazy 

logic that inferred, from the failure of already ambitious research/ a need 

r . * . 

for even more ambitious research. , 

One result of that logic was to debase the modest jobs that social re- 
search can do well, by setting unattainable tasks that social research oatinot 
perform. Another result was to needlessly magnify the failures of applied 
research;' by the end of the 1970s, educational^ R$D had produced unimpressive 
results all across a large menu of impossible tasks. This did nothing to im- 
prove the reputation, either of social science or of its applications. 

Still another result of the growth of applied research in education-- 
o% perhaps a cause of it--was a tendency to confuse .scientific analyses of 
practice with what teachers and administrators actually do. The odd idea 
grew up that through social science, federal administrators can know enough 
about educational practice to make great changes in the work, of local -teachers 
and administrators.* This notion lay at the heart of the Experimental Schools 
Program, among many other activities in'the last two decades. Federal managers 
believed that applied research could actually reform - the knowledge 
that local teachers- and administrators used in their work. ESP 



managers, for example, believed that if they devised a nerw and more compre- 
hensi^e^scheme for social science evaluation, reliable scientific information 
Could replafcw^the knowledge commonly used in managing school districts, local 



schoolii^and classrooms. They ignored the frailties of social science, 

and ignored as well the strengths of both ordinary knowledge and the accumu- 

* 

lat^d ^professional knowledge of teachers and administrators'. Like many other 

federal reformers of.the last t4vo decades, ESP managers assumed that social 

science information can provide authoritative guides to action, .and that 

these can strongly influence what teachers and administrators do. 
4T • \ 

\ But social -science, while useful for many things, could not bear the 

weight that federal administrators tried to place upon it. The ESP evalua- 
tion was a resounding and ^tJstly failure. It'was instructive, though, for it 
encourages us to to ask whether, among practitioners^ many resources in edu- 
cation, scientific knowledge can play more than a very modest role. 

However excellent the studies, can educational practice depend heavily on 

■> 

social science? Can government use social science as an effective 

'.instrument of persuasion? Scrutiny of the Experimental Schools Program can 
'illuminate these questions, among others. 

II. ] FEDERAL VS. 'LOCAL APPROACHES TO CHANGE * V 

The aim of ESP was comprehensive change. This was seen as a new way of 
thinking. about how to change schools, in contrast to other federal programs 
tha^t .made what were viewed as "piecemeal 11 or . "categorical" change (e.g., 
Title I).* The notion of comprehensive change is familiar in federal 
"sofcial policy, but the federal ESP staff saw it as a novel approath 

to reform" .in ^education. An internal ESP memo explained that piece- 

* * " * 

moal efforts?; such as "curriculum reform projects, failed because the 



(new! curriculum presupposed know,ledge\or skills' npt gresent in teachers 

who wete' not given adequate training and the curriculum as intended 

never was implemented . ,l3 The thrust -of the criticism was that there was 

' little coordination between two elements 0 f a school system, teachers 

and the curriculum, and so reforms failed. Another memo explained that 

new programs M were introduced without adequate involvement and/or prepa- 

ration of a community which raised the normal level .of anxiety associated 

wit>h any change to the point where any problem encountered in the imple- 

mentation, of 'the program could be sufficient "to kill it." 4 The implicj> 

tion, was that such programs ha"d failed because they did not include one 

of the components considered essential to coherent reform: community 

participation in the local education agency (LEA). * • " 

I * 
The ESP staff thus believed that significant improvements in edu- 

cation wers unlikely without "holistic" or "comprehensive" change, 

A " • - 

i.e., unless the different "practices"^ separate "pieces 11 were com- 

' : - ' 5 • ■ * 

pa^iblq and mutually reinforcing in a "synergistic" way. They further \ 

assumed ^that a school system can be neat¥^;described,*and madfe to, work # 

coherently. The £SP description explained that change niust "include 

but not be limited to 1 ' the "five major 11 parts of a school system: 

"curriculum development, community participation, staff development, 

6 * 

administration, and organization"; if there Cwas coordination across 

these five parts of the system^ coherent' cH&nge would result. Based 

on this assumption, the federal. ESP Ijtaff required school districts 

** . 
to define their plan for change via arfr "educational concept" or 

^2^ejtf;Fa4^hei?^-^pre^ plans- -to- -get froniT^what 



exists at pr^gent to what education ought to be in £erms of 'its needs * 

7 * 
and aspirations."- . \ 
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The Local Response •■ r ' ', *' 

Lacal approaches to the ]*rogram c}iffered> substantially fro/n federal re- 
formers 1 hopes. As one federal *' monitor reported, local* sites, 
especially' those in rural areas, could 1 not grasp . the meaning of* com- 
prehensive change. In, responste, federal monitors »of ten treated 

local personnel "like graduate students who had a dissertation to ' 

\ 3 

j defend." Another federal m6nitor touched on the heart of the dif- 
ference in federal and local conceptions of change, when she said that 

9 1 

local. projects "refused to think of pieces as interlocking parts." 
Local practice simply did not reflect the coherence assumed in the 

federal notion of comprehensive ch^ige, 

v % 

* Problems at the Administrative 

Local administrators typically interpreted comprehensive change 
i ^ 

in tefms % of local needs and operations. One superintendent > 



c«.iqvolved in curriculum reform explaiwd that he was attracted to the 
h program because it "looked broad. enough so that we could' attack any 
- • .curriculum problem we could find." He delated comprehensiveness only 
to' curriculum; and he chose curriculum from some sense of its com- 
patibility with parent work in the school district. He went on to 
say that "what is comprehensive in math is that it. is K-12, what is 
comprehensive in history is that it is community-wide in developing 
(sic)." ,He was perplexed and perturbed, like others in his 

position, when ESP Washington rejected his proposed program. He summed 

s 

up the problem by saying that the Washington program* director was 
"hard on comprehensive "change . . » and no matter what ,we told him it 
was never comprehensive • • .He did not want us' to just t>uy text- 

ERJC ; , . ' K • IS 
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books, he wanted us to think bigger. 11 The superintendent was never sure* 
what "bigger" was. At the local level, 'changing the curriculum seemed 
a sweeping and "comprehensive" reform program. From the federal perspec- 
tive, however, change in all five areas, not just curriculum change, seemed 
essential. 

Local administrators in another site ran into similar problems. 
Asked what he understood by comprehensive change, the superintendent 
replied that he "blocked on the word" because he had "spent so many 
months at it." He added that the "feds" kept demanding that the dis- 
trict define comprehensiveness, but that "the game [the district] got^ 

into was a series of Title III projects." 11 -These were piecemeal 

•> 

educational innovations such as individualized curricula or supple- 
mentary service centers. They -reflected little joint decision making, 
for the local project directors confined .themselves to their own projects 
This was precisely what federal reformers &id not intend: instead of 
a single coherent program of district-wide reform, they found *ESP supporting 

§ 

piecemeal local projects, running independently of one another without * 
common focus. 

* •«* 
In another site, the superintendent was preoccupied with expanding 

existing federally funded local projects. In proposals to ESP^ he sought to 
connect a, number of these by explaining that "comprehensiveness" lay in the 

V 

•* 12 
"optimum learning environment" of the school.- ft But the connection between 

/ 

projects was defined only vaguely and retrospectively, and involved 9 
little decision making across projects. His newsletter elaborated on his 
interpretation^of comprehensiveness: the "proposed [ESP] project could mean 
that many federal projects now operating . . . would be expanded into an 
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t ^ 

I operational district -wide se.tup, The stress is no,t on developing a new * 



program but upon operating the many programs now established throughout 'the 

. ' , . . ; 1/3 
district/ 1 

Once again, then, the local definition of comprehensive- change rested 
/ on what was already occurring in the schoo'l district. Like most definitions 
offered, it was confusing, One reason for this is that the concept itself 
is indistinct. Another is that local administrators attempted to unify un- 
related activities by means of words designed to satsify federal monitors. 
But if the words portrayed a broad program pf integrated change, the loc&l 
projects actually consisted of incremental modifications of existing projects^ 
little related to one another. One superintendent discussed his view oV ESP m 4 
Washington's negative reaction: he felt that the project director of the 

National ^institute of Education (N1E) "harped" on the fact that his proposed y . 

. * 14 

program was not "comprehensive." He saw the expansion of activities across ^ 

the district as promising comprehensive change because it touched on so many 

areas of practice, d^s jointed*£hough they were. The federal staff, 

by contrast, 'saw his program ^f^n attempt to make disconnected #> 

projects appear to be a coherent whole. 

Conflict over the meaning of comprehensive change was thus a persistent feature 
of the program. Local administrators continually proposed plans to incrementally 
/ modify varied and often unrelated existing endeavors, while the federal staff 

wanted a4JL_ endeavors to reflect an overall, consistent, district- wide 

f 

plan, and a big change, in the way of doing business. Having won , 
ESP grants primarily because they had shown a willingness to make 
comprehensive change, l^ocal sites could ^produce no specific plans to 
that end. for one thing* local operations themselves had not the 
coherence that would Support such a plan, , Moreover, local managers ' 

ERIC . ' 20 ' ' * 



were unsure what comprehensive. Change was. They proposed and reproposed 
programs of change"-; the federal g$P rejected them as lacking the coherence 
required for comprehensive change. As ong superintendent .reported, after 
rewriting his district plar\, several times he "did not know whether [it] was 
finally right or not, but I guess it,, was because they approved it." 
Another project director, asked about comprehensiveness, explained, "We 
never did resolve that damned thing, we never had any agreement on what it 

meant. In at least one case, a local si'te refused to rewrite its plans 

17 

aftef several attempts. As a* last resort, the federal ESP staff often 

f 

just approved the most recent plan, hoping that local understanding of 

a comprehensive approach might* grow as the program proceeded The program 

had to be maintained, even if local projects were "wanting. .» 

/ 

The^difference in approaches to "change was also manifest in local 

/* 

sites* inability to off er acceptable, central themes for th'eir projects. 

1 • ' 

The \federal staff intended that theme to be a common basis for changes in 

\ < 

different parts of the* school system, y^cpjiformity with their assumption 
that Comprehensiveness would asstrre a new' coherence in local practice, Y^t 
a c^nti*Ll theme consistently eludj^ct local practitioners, ^Administrators saw 
potential change occurring not in accordance with any single theme, but 
ratheA in various ways in different parts of the school system. One local 
project director, in explaining federal rejection of his plan, confessed 
that he "did not know what [the program] was all about, except there was 
supposed to. be a theme for the project," He went on, "Every time we came 
up with something, [the federal monitor] did not think it was what it was 
supposed to be. And I just did not know what she wanted. Every time I ^ 
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submitted theme, one sentence mind you, she rejected it 

The federal ESP staff saw proposed themes as either'too vague (e.g., 
"individualization") or so specific (e.g., curriculum reform) that they 
ran counter t6 the notion of coherent change involving the five elements 
of- the school system. To local administrators , , the requirement for a 
central theme was a contrived rationale for different programs and acti- 
vities only tenuously related to One another--an academic exercise. To the 
ESP Washington staff*, the theme was to indicate the main direction, of a 
district-wide program of change. ' ^ p 

Ironically, local incoherence was to some extent .fostered by 
federal managers. Local ^administrators used ESP to support various 
specific efforts at change, because they were pressed to support many 
different activities and. projects in their districts: typically there 
were more local requests for project support than could be supported.. 
Yet federal ESP st^ff kept exhorting local administrators toward broad * 
community involvement in the change process, because they believed that 

this would promc/te coherence.. Federal officials did not seem to realize * 

\ 

that, given the considerable autonomy of sub-units in local school systems, 
the greater the number of local participants involved the more disparate 
and incoherent the program would become. One .superintendent commented: 
"The [ESP staff in Washington] kept saying that we were not opening 
it up enough, but we- thought we had opened it up too much." 19 For 
the local administration it was a Ca.tch-22 predicament: the more 
accessible ESP was made to the district, the more diverse and numerous' 
we re the requests by thos e seeking support for their particul»-f*e~--- 
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grams. Thus as local administrators' complied with federal demands 
for greater local participation, the program was increasingly shaped * 
by local groups 1 diverse interests. 

Problems at the School Levei . , w 

Some of the reasons why local sites did not perform up to federal 

expectations can be seen in principals' and and teachers 1 responses 

to v ESP. Principals within school districts had diverse 

conceptions of ESP, only loosely connected to those of central admin- ^ 

isttators. In one site, where the central administration's "theme" 

was human ism^and individualization, a principal explained: "We kept 

getting the words o^ humanism and individualization, but I think 

ncbody had the. same meaning about what we meant. "^ A principal in 

another district explainect - that for him or anyone else^ ^ the 

central administration's notion of the program was H hard to say 

in a detailed sense without lots of false statements and misunder- 
21 

standings." 'Another principal recalled, "It soon became obvioij^ to m 
that (the local project director] did not know what was going on and 

the other principals did not know either. * So he would keep asking 

* 22 

us [what our ESP project was about] because he did not know." 

The diverse ways in which principals within a school district 
thought about change were due partly to the fact that different 
changes were desired in different schools. Operating largely inde- 
pendent ly o*f one another, principals had little reason to want to 
understand, let alone to rely upon other principals' thoughts about . 
ESP. Their interest was limited to* what was feasible in their school. 
As one principal remarked, "At first I thought the project w$s 
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offering a lot of work I did not need. But then I realized that I 
could do Lots of things I did^t see before. 1,23 Another principal 
^ said that he saw the program "as an opportunity to develop different 
programs we could not get from other federal agencies . "^ Typically, 
local change was seen only in terms of a principals own school and % * 
was thus inconsistent with federal notions of district-wide "syner- 
gistic" efforts at "comprehensive change." 

Teachers' reactions were similar. Often they ^ 

didn't even know about comprehensive change, despite the efforts of* 
some principals and central administrators to explain the, program. 
One senior federal monitor, after visiting a highly regarded project 
after a year x>f operation, reported disdainfully, "My speech to the 

4 

teachers regarding the central theme of experimental schools and of 

the local concerns clearly w^s the first time they fiad heard of com- 

25 

prehensiveness . " In addition, teachers 1 views of £SP generally. 

had little to do with what central adminis- 

trators or principals had in mind. The program often # 

was o^ no interest at all. i One teacher commented, - 
"I'm not into this alternative thing [the 'theme 1 of her site]; that 
is theit bag. I am into the classroom and do my own thing there."** 

Another was "sure good things came out^of it," but could not identify what they 
27 

might be.* And among those teachers who thought they were working in ESP--many 
% didn't even know it-- each one's activities seemed little related to what 
other teachers in the school were doing. As one teacher explained, 
"From what observation I have made, I have -seen as much variety 9 
going on in one school as there is in any single program- [of school- 
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wide* change] ," But since ultimately the change took place at the 
classroom level, in accordance with each teacher's interpretation, e\ on 
school-wide change often proved ^llusorV, According to one ESP project 

staff member, "Everybody worked lilj;e hell—like horses fulling in 

29 

different directions." 

Some inertia was certainly involved. Teachers wantefl to con- 
tinue what they were cjoing before ESP, and they did. 

m 

As one teacher put it, "For the most part, everybody protected the 
status quo . . % after the initial thrust everybody protected what 

.they had. ,|,>0 Others were hostile or simply apathetic toward^change 

efforts initiated by central administrators or principals. One teacher 

described the feeling in her school as wanting to be left alone", and 

31 

was herself, "downright negative" to ESP. A curriculum coordinator 
explained, in rather typical fashion, what happened when the new cur- 
riculum was brought into her school: "Some of the kids were going to 

get, a self-concept and self-awareness (sic) so they brought all, sorts 

32 

of material but little came of it," A classroom teacher said sar- 
castically that of several new curriculum reforms tried in her school, 
"The one that had the most success and made the biggest single con- 



tribution was theTullman Manual for Physical Education?"*^ She 
meant that while only a few teachers used the new curriculum, the 
Pullman Manual succeeded because there was only one physical education 
teacher, who used it. 

0 

Problems at the Federal Level 

The unexpected variety of images and interpretations evoked by 
ESP at the local level wa s troublesome for federal managers, Soae 



ESP staffers couldn^t figure out whether various 

t 

activities in a local site were related, or "comprehensive. 11 Nor did 
fe4eral reformers agree among themselves whether what was occurring 
in asite was significant or not--or in some cases whether anything 
was happening at all, After a site visit, one federal monitor com- 
mented upon another monitor's perceptions of a prizecU- 
site, believed to be successfully making significant comprehensive 
change. iier memo ex P la ^,ed that, contrary to her fellow monitor's 
perceptions, "the local project certainly was' not comprehensive and, 
further, no one involved in writing the plan, executing it, adminis- 

34 

ter4ng it, watching it, was aware of what comprehensive change meant." , 

f - 

Sh'e concluded -£hat in this site ESP had produced little change that was 
not occurring before the program 

Conflict and confusion were not limited »to the staff. 

* 

Even the first director of NIC, after assuming responsibility for 

0 

35 

ESP from the U.S. Office of Education, was unclear about the program 
* 

and the meaning of comprehensive change. In an interview he stated 
that he had never understood "what Binswanger [the ESP director] was 
getting at with his notion of comprehensive change." It struck him 
as "more an experiment in ^government funding and intervention tech- 
niques rather than an experiment in educational concepts and tech- 
niques. f - r He further explained * that the program had already failed 

when he took it over; and that it continued becfluse , as with othefr „ 

i 36 

programs of reform, terminating it was aj'political irnpossibility." 
Nonetheless, ESP program staffers held to the view that 
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ESP failed for the same reason earlier reform efforts had failed: 

local resistance to federal plans and' intentions. Local 

sites, they held, were unwilling to change as their central adminis r 

trations had at first promised. Instead, according to one federal 

monitor, local sites had "done what [they] needed"-- that is, used the 

37 

program to meet local needs. Others saw local sites simply as 

"ripping off" the government, or engaging in activities that were 
"fraudulent, " % As one senior federal monitor said, reflecting on 



the program, "I would do it again but with [local] people who believed 



in it. ,r 
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Interview-with school principal in the -Pranklin Pierce School District, , 
Tacoma, Washington, March 30, 1977. 
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Interview with English department chairman in the Southeast Alternative 
Schoo^ District , Minneapolis, Minnesota, June 1, 1977. 

Interview with local ESP project staff .member in the Edgewood Independent 
' School District, San Antonio, Texas, April 12, 1977. 
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Interview with teacher in the Southeast Alternative School District, 
Minneapolis, Minnesota, May 31, lf)77. 

"^Interview with teacher in^Supervisory Union #58, Groveton, New * 
? * Hampshire, June 9> 1977. i 
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Interview with school curriculum arts coordinator in Supervisory Union 
#58, Groveton, New Hampshire, June 9, 1977. 
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■ ^ Washington, Match 30, 1977. 
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ESP was transferred from the Office of Education to the newly created 

National Institute of Education (NIB) a year and a half after its 
inception. For a more complete discussion of the effects 
of the creation of NIE on programs, see Sproull, L. , et al. 
Organizing Anarchy: Belief, Bureaucracy and Politics in the » 
National Institute of Education . Chicago: University of Chicago 
Press, 1978. 
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Interview with former director of t the National Institute of Education, 
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Interview with former senior federal ESP^ staff member, November 17, 1976. 
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III. FEDERA'L VS. LOCAL UNITS OF PRACTIC E - 

-One essential part of the ESP idea was to insure local commitment by 

'I ♦ 

insuring local control. The federal managers assumed that if local adminis- 

trators felt that the program was their own, they would do a better job. 

The federal ESP staff saw their primary task as giving Control to local 

administrators, and holding them accountable for implementing the programs. 

This, of* course, assulted that comprehensive change in a district could be managed 

1 . . ' 

in a centralized way—that control lay in tlje central administration. This idea 

* . \ < 

made a certain kind of sense for federal practitioners, for the unit of their prac 

' tice was the progranf?* how could they deai with each local school individually? 

But the control of local central administrators is limited: the units of lpcal 

practice are schools and classrooms, yhich are at least s emi -autonomous , 

Local administrators cannot manage school districts without^ejnir«ing a 

great deal of autonomy for .teachers and principals. In managing the ESP 

program, they followed this pattern, delegating control to the school level, 

and acquiescing in principals 1 and teachers 1 plans for change* In the ,° 

decentralized A organization of local practice, * the operations weeded 

to insure the interest and participation of the local units conflicted 

with the. federal assumption that .district-wide change could be centrally 

managed. * 

Local Control as Federal Policy 

1 At the time of ESP's inception, fostering local* autonomy was encouraged 

in federal policy circles. The previous decade's experience had .convinced many 

federal reformer^ that, simply "laying on money in the nafne of 

2 

doing j gooc^ did not . solve social problems. Moreover, local 
educators ,had been complaining about the in^fensitivity and domination 
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of the f ederal^Sureaucracy . Federal reformers were accused of alien- 
ating local school practitioners and suffocating local commitment in 
their programs for educational change. The federal director of ESP, 
Jtobert Binswanger./^was particularly aware of tnfese co^cerns^-and-com-- . 
mitted to local control. One of his colleagues explained that it was 
this aspect of ESP which "turned [Binswanger] on." Binswanger wanted 
"to have a program which was really responsive to local schools-- to 

give a good idea a try--not that it would necessarily work, but ^to try, 

Jr 3 K 
... r s," Tom Glennan, the 

first director of NIE, explained that ESP had emerged from a Congres- 
sional climate in which federal control of local programs was bJing 
severely criticized: "Schools should be able to do what they want." 4 
Another member of NIE, formerly on the ESP staff in Washington,, said 
'that ESP w,as conceived at* a time wtfen any concept that included local 

i f 
5 % 

control w3s. popular. * . 

ESP program staffers. believed tha# local control was consistent witn 

comprehensive local reform. They simply assumed that there was a locus of 

control from which LEAs could be directed. For example, when ESP sought out 

local districts interested in a "comprehensive" reform program, 

recruitment concentrated only on school superintendents* ^ They were 

seen as the key to local action, and their state of mind^was regarded 

as critical. The views , comments , and wishes of principals and teachers 

were not even explored. , . 

To insure their administrative control, BSP staff proposed 
j . f . 

a "rigotous 1 * system of Monitoring, while emphasizing that such a 

* * - 6 

system ••must respect and support the local project design." ESP 
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monitors were to work directly with local central administrators in 

> 

a "partnership." It remained unclear how federal partners ^were 

to respect local contr&l if the project did not conform to federal I 

expectations. $ 

Local Control vs . Federal Ambitions 
e * = 

Earlier federal reform programs had been plagued by a lack of local 
commitment to federal program plans, and by consequent attempts tiy federal 
executives to changfe local programs. Although ESP wa_s des s |gned 

to overcome this problem, it didn't. Tf\e more zealously federal reformers 
tried to encourage implementation of ESP, the more they interfered with 
local autonomy, and the greater was local resentment. As one superintendent 
remarked, "[The federal ESP staff] were attempting to take complete 
control of a school district through an -ESP program by their own 
methods without regard to the wishes, and practices of the school 

-7 

district. f,/ 

There was conflict between the federal ESP staff and local prac- 
titioners from ' the outset. latt ^ r were understandably # 
interested in .obtaining' - grants, and to this end wished to please 
ESP Washington. Some had received planning grants based on brief initial 
letters of interest. But the proposals they then prepared under these grants 
were more serious than the letters, and had to fit With the 

/ 

local organization. This made it difficult to achieve the compr^e- 

hensiveness and ^coherence ESP waited. As one superintendent explained, 

thedistrict "wanted, it to be our prdgram, but we could never write 

anything acceptable to [ESP Washington] . M Eventually, after several 

♦ J 

frustrating encounters with the federal ESP staff, this superintendent 
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simply called the federal project monitor so that she "could tell us 

8 ^ \ 

the words she wanted us to say." This, of course, involved at least 

a nominal loss of autonomy, as it elimingtted control of project design/by 

local participants. A project director in another site said of his proposal 

writing: "My problem was that-thk more they demanded . , . [compre- 

i hensive] change, the further it took ownership away from those who 

developed the pr/ject; and the more they asked me to rewrite, the more 

isolated I became from the $tpiff, because I was really bastardizing 

9 ^ 
their work. " . a 
* • 

Here and elsewhere, principal! and teachers began to resent 

r 

local' administrators, because of what seemed a loss of autonomy-. 

c/ne principal explained, "Like most teachers, I kind of felt that it 

was a program beings pushed on them." 10 And federal pressure often 

placed local administrators in a position in which they had to make 

their projects appear to be something they are not. 

As one project^ director put it: "Binswanger had a way of telling us 

what we would want. . , . [His] thing was to get the district to f do 

what they wanted, 1 yet to get the mon^we had to lie," by making 

the district seem more coordinated than it really was . 1,11 

% 

\ 

The problems did not diminish as the pragram took shape. 4 ks 

j + 
one-local project director said, "We thought it was going to be no 

government interference and we cowitf^do^what we wanted with the 

♦ 

# *• *i *y % 

^ money. Instead there was constant harassment and- interference." One 

superintendent was initially enthusiastic about what he described 

as the government's notion of "non-interference" in development- of 

his program*. But after "he went *to Washington— as he thought, to sign 

'. 34 • . i .- ' j 
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the agreement for funds—he reported: "(EpP Washington] tore our 
budget apart while we sat there completely unprepared. They threw 
out this and that and tried to tell us what we needed tp do for the 
money.' 1 But what Washington told its local partners often did not 
fit local plans for reform. Federal insistence on community involve- 
ment, for example, meant more demands on and less control for local central 
administrators. As one superintendent explained, "One of the 
typical problems with the program was that the federal government 
wanted the community involved, but the community represents such a 

diverse population that its involvement only created different factions 

14 

of interest who wanted different things to happen in school." 

In one site, in fact, -Federal o^icials saw, the failure of local 

administrators to involve the community in proposal writing as a prime 

problem. The superintendent explained, "NIE kept pushing the community 

school idea." He felt that they had already involved the community too much 

* 

whereas "NIE kept pushing for more."^ One federal E8P staff member, 
after a day of argument with local administrators over their initial 
proposal, wrote his own proposal outline. Local administrators fol- 
lowed that outline in # the ne'xt proposal they submitted, and it was 
approved shortly thereafter,^ But what "their" proposal now: prom- 
ised was no longer what they had intended. In another case, the 
federal ESP staff wanted every school in the district to be iricly^ed.' 

in* the program, but, as the.superintei^ifent explained, there were 

_ « 

great difficulties "in trying to get different schools together, 
because every school wanted to do their own thing*" Federal insistence ^ 
.on local involvement reduced local administrators* ability to initiate 
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change where they thought it might succeed, and reduced central control 
even more. 

LQGal administrators were often bewildered by federal 
✓ » 

efforts *to shape their projects. According £ 

to one superintendent, the ESP staff "cut some components" of local 

projects "because^ they were not in some way, a way I cannot define, 

18 1 
consistent with what NIE wanted." And a local project director 

reported: "We. would ask [ESP Washington] what to do and they would 

say, 'you determine it.' And then after we would, they' wo,uld say it 

19 

was not right and not give us the money. It was really a-^td^-22 . " 

Loc'al aataininstrators felt that their autonomy was being 
compromised. They were being told to develop their own programs at 
Xhe same time-as federal reformers pressed them to meet federal notions 
about how the' r change should occur. Washington consistently demanded 
a coordinated program that ran against the' grain of the decentralized 
local organization. And the tension betweer\ federal and local staff 
was exacerbated because federal administrators seemed unaware of this^ 
divergence between federal and local practice. 

These conflicts arose plainly from the , 

federal assun^tion that change could be centrally managed 
in localities,. Yet even when the problems of this belief 
became apparent, federal reformers never lost their conviction * 
that ,the local centril^dministration was the key to program 

coordination.* This gave ris,e to further problems, for federal ESP staff 
thought it followed that, if control was not being adequately exercised, the 
local administration pas responsible. Thus if reform did not live up to' 
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expectations, the logical next step was for federal practitioners to 
tinker with the local administration. This they did. One super- 
intendent related how his appointment of a project director, who 
later became a HEW division director, was vetoed by the federal j?ro- 
gram staff. He was tola that his choice was rejected because "she 

had conceptualized the program and was not able to implement it in 

20 

an objective fashion. 0 According to the superintendent in another 

site, the .federal monitor tried to secure the resignation of the 

project director, to whom NIE attributed difficulties with the pro- 

21 

gram, by threatening to withdraw funds. And the federal staff 
tried to influence local administrative organisation as well as 
staff. • One superintendent reported that the federal program 
director had ordered what the superintendent described as "reorgan- 
izing the whole administration. M His response was, M l was the 
[superintendent] and I was not about to change our organization. " 
Not understanding the reason for this intervention/ he saw it simply 

as a direct contradiction of promises of non-interference by the 

% 22, 

federal program director. It certainly, seemed an odd way to pro- 
mote local control. 

Local administrators often did not share federal monitors 1 views 
of the program, and they . did not believe that federal ob- 
jectidns could be met through federal intervention in the school 
■ district. Nearly all local administrators saw their authority 
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being undermined. As one superintendent explained, after he had * 

been asked to remove the*original project director: "On the bright 

side, it had tffe effect of unifying the principal and the staff of 

23 

[the district] because they felt they had been done in by NIC," , 
but it did not enhance local commitment to v the ESP projec^. 

Federal actions thus belied ESP assertions about commitment 
to local control. Some local administrators gradually aban- 
doned their attempts to maintain control of the program. Where 
federal monitors were active, local control tended to erode/ 
and with it local commitment. Federal practitioners, for their part, 
would not or could not accept the decentralized organization of local 
school districts. They saw the lack of coordination as the result of 
weakness of will, or intelligence, or both. The federal ESP staff saw 
ocal control as a desirable reform of past federal practice, because it 

/ 

vjould promote local commitment to comprehensive change. But control at 
1 he local level is decentralizedT While the Washington ESP staff believed 
that change could occur at a superintendent^ direction,, local teacher^ 
and principals exercise much control independent of the central adminis- ^ 
tration. Managing a local district consists in large part in accommo- 
dating these diverse local interests. As a result, ESP, liJce other federal 
education programs, took a variety of local shapes. But the Washington 

L ■ ' ' ' l • : 

• staff saw this as a failure of local leadership, while local adminis tratoys 
saw it as an essential feature ^of their work. 

Late in the life of ESP, one js^ftw federal feSP staff member reported 
that she sti-11 saw nothing i^ong w±tfi the notions of comprehensive change 
and lo<^l control— the problelh was in local implementation* What ESP * 
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had needed, she said, was more sincere, dedicated, and capable local 
schooj staffers. 24 
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Footnotes to Chapter III 

*This may seem a necessary belief for federal practitioners. Tolstoy, 
in War and Peace , discusses Napoleon's ,! cominand f! at the battle, 
of Borodino: "And it jvas not Napoleon who directed the course 
of the battle for none of his orders wer^ executed and during 
the battle he did not know what was going on before him. So the 
way in which these people killed one another was not decided by 
Napoleon but occurred independently of him in accord with the 
will of thousands of people who took part in the common action. 
It only seemed to Napolebn that all took place by hi^ will." . > 
(Tolstoy, L. War and Peace . New York: Washington Square Press/ 
1963, p. 399.) Administrators, like generals, may command more 
than they control. 

2 

Timpane, M. Educational experimentation in national policy. Harvard 
Educational Review , November 1970, 40(4), pp. 547-^66. 
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Interview with federal ESP director of evaluation, July 29, 1977. 
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Interview with former director of" the National Institute of Education, 

October 24, 1976. 
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^Interview with federal ESP staff member, October 29, 1976. 

^Internal federal ESP program document prepared for the National Institute 
6f Education, dated September 18, 1972. - , 
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Interview with former district superintendent of the Edgewood School 
District, San Antonio, Texas, April 13, 1977. 
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Interview with district superintendent in Craig City, Alaska, May 4, 1977. 

9 - 

Interview with local ESP project director in the Constantine Schoal District 

Constantine, Michigan, May 2, 1977. 

10 ,* J 

IntervieW with school principal in the South Umqua School District, 

South Umqua, Oregon, May, 24, 1977. 

Interview with local ESP project' director in the, Berkeley Unified School 
District, Berkeley, Califorh^a, May 10, 1977. 
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Interview with* local ESP project director in the Carbon County School 

District, Sa ratog a, Wyoming, April 21, 1977. ' 



13 Interview with district superintendent of the Carbon County School Distinct, 
Saratoga, Wyoming, April 21, 1977. , 

m \ * ' 
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Interview with district superintendent of the Berkeley Unified School 

District,- Berkeley, California, May 11, ^V 1 -^^ 

15 Interview with district superintendent, oy theF^anklin Pierce Schaol District, 
Tacoma, Washington, iiarch 30, 1977/ { 

16 ,ibid. i ^ 
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Interview with district superintendent of Supervisory, Union #58, Groveton, 

New Hampshire, June 7, 1977. 

18 

Interview with district superintendent of the Franklin Pierce School District, , 
Tacoma, Washington, March *30, 1977. ^ 



Interview with local ESP project^feector in the Edgewood Independent 



School District, San Antoifl^Texas , April 13, 1977. 

Interview with district superintendent ^igjfe Edgewood Independent /SBwol 
District, San Antonio, Texas, ApriTl3, 1977." i - J^:^ 
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Interview with district superintendent of the FrankFin Pierce School 

District, Tacoma, Washington, March 30, 1^77. „ 
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Interview with district superintendent of the Carbon County School District, 
Saratoga, Wyoming, April. 21, 1977. „- * 
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^Interview with district superintendent pf the Franklin Pierce School 
, District, Tacoma, Washington, March 30, 1977. 
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Interview with senior federal ESP staff member, November 17, 1976. 



IV. KNOWLEDGE AND PRACTICE 

Federal school reformers believed that change could be 

greatly facilitated by new knowledge. They considered the everyday 

knowledge and skill -of local practice to be deficient, and thought 

that scientific knowledge could help to improve practice. The notion 

that social science would be useful to local practice and federal policy 

makers wis enthusiastically accepted in the late 1950s and 1960s, 

* But efforts to improve federal^programs and local practice tffrough 

social science consistently fell short, both in providing information 

helpful to management and in determining the success of programs. Thus, 

the 1960s were. parked by repeated efforts to improve the application of 

social science to education. The ESP was one such improvement. It was 

cited in HEW Secretary Robert Finch's opening comments to the House' 

Committee oh Education as one of several programs that were to address 

-j 

earlier scientific shortcomings. 1 ofte year l^ter, President Nixon,' in 
his message to Congress on "Reform and Renewal 11 in education, described 
the. ESP as the "missing bridge between educational research and actua* 
practice. 11 The ESP was seen as the prototype of research programs to be 

2 y* 

contained. in the newly proposed tylE. * 

The 'ESP staff saw their mission as part of NIE, ! s proposed response « 

to the ^general criticism 11 of educational reform that had bee^\ "extended 

3 

to include previous^search . . . The B ESP staff reasoned that past 

evaluations "had^failed to capitalize qn the opportunity to*contribute to 
the knowledgebase" because they jused traditional social science methods: 
they typically looked only, at outcomes, and found "no significant differ.- ^ 

ences." Being Ignorant* about* what had actually occurred in school pro- 

*- ' ^ ' l I ' ' 

grams, these evaluations offered little useful information on the change 
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The ESP staff argued that the deficiencies in program evaluation 
could be rectified 8y a different' and stronger commitment to social 
science. This was apparent first in its apportionment of 2S°6 to 30% 

j>f the program funds for evaluation,^ Second, this evaluation, 

I * 6 

unlike others., was not to be conducted on a "fly-in fly-out basis. 11 

It promised to use and develop "new and improved research and eval-/ 

uation strategies 11 that would be closely attuned to the process of 

change in schools — how it worked, why it went well or badly, and 

wh£t helpfed or hurt. This in turn would allow; "(1) significant 

improvement in .the knowledge base concerning formal education; 

(2) better understanding of the processes of change in education; 

^3) more satisfactory investigation into the effectiveness of 

schooling; (4) the generation of programmatic information that will 

7 

be of immediate use to practitioners." There was a pervasive be- 
lief that local practice could be^refined by supplanting or supple- 
menting practitioners 1 knowledge and skills with social science, and 
that this could be achieved by using scientific techniques that were 

"innovative 11 and "experimental in conceptualization, approach and * 

* • 8 

practice." How the evaluation was to be conducted was less clear; 

, • Ty • 
but that social science was the appropriate tool with which to reform 

local practice was, aggressively asserted. 

Two characteristics of the evaluation showed the direction of 

intended reforms, "First, the program required a research .vehicle . 

(which would maximize whatjone could [learn from a .collection of field 



l 

experiments-- summative evaluation and documentation (that is, know- 

** * * 

ledge about the degree to which •comprehensive change 1 was occurring 

in any school district); and second, the program required the support 
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1 of an evaluation designed to assist in the improvement of programs 

, and projects while they were in the process of development and 

9 * 

'operation—formative evaluation % " The evaluation of ESP was to have 
two related reifcjrni crit^xiru: the innovative methods to bemused and 
the new ty^^>f information to be produced; and improved management 
of federal programs at the federal and local levels through the use 
of program-related information supplied byESP f s evaluation. 

To that end, an elaborate three-tiered evaluation scheme was 
proposed, The major evaluation component (termed "on-site" or "level 
II !t project evaluation^) was to be conducted by private firms of 
professional social scientists, It was to account for two thirds 
of the^monies spent oft the evaluation)* and to address the/main fed- 
. eral concern : to improve local practice by producing a new type of 
socia)&sci;ence knowledge on the process of local cltange, which would 



3 heUfi- improve program management. 

The* ESP staff was ^particularly concerned with past, social science 

v ' .* . i 

efforts to determine the ' criteria 'and judgments about the outcomes" 
# * 

of schooling. "Many studies, 1 ' an early ESP staff memo stated, "can 
;bc criticized on the ground that they h£ve equated measurement of a 
relatively narrow band of cognitive skills with the outcome^pf 
school ing." To^ address this concern, the ESP evaluation sought 
"alternative strategies -for research" that would "not only have to 
^develop broad-base^ measures," but would also "have to stud^par- 

ticulaf^phenoraena over sufficient time* and insufficient depth* to 

* * / • 
allow-for/mdne costly methods of behavioral observation as well^as , 

■ t ✓ * . * 

, paper and .pencil tests," The -€SP program* staff -was also concerned 

* t • 

that "variables which are traditionally studied have not been adequate 
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.in explaining differences" in l<|cal school change efforts. It was 

i 

therefore proposed that ESP develop a research strategy having ,f a 
h . ' 

different and probably more qualitative s-et of variables to measure 

* < 
differences between schools,^ By so doing, the ESP staff thought 

they could' figure out what made change work, and thus how to 

improve local practice. 

The information to be produced by the level II evaluation was 
•therefore to be useful to federal and local decision makers. The 
ESP staff held that if the evaluation, which sought to serve decision 
makers, .was accurate, then it would ipso facto be useful to decision 
makers. But this assumed that social science knowledge would b^e .pref erred 

to the knowledge that was ordinarily used in local decision making. It 

was simply assumed that social science knowledge was superior to ordinary 

or professional knowledge, and thus would be used in decisioit making. 

The second way in which social science knowledge was to improve 

local program management was through ,f level I" 'or "local project 11 '* 

evaluation, "performed by or* under the direction of local project # 

personnel" and complementing the level II effort. It constituted a 

small part of local site budgets and was \o be "formative 11 -- i t e< , "oriented 

primarily toward" operational decision making and planning <"* 3 Level I 

evaluation was to satisfy local project needs "to impfove the ongoing 

14 ' 

operations of ,a particular p/oject*" Apd it was to minimize the political 
controversies that had arisen over evaluations in the past. 
An ESP »eao explained, "There is^a natural conflict between the needs 
of a long-terla st ( uUy and the needs of project participants for imme- * 
diate help, therefore if one wishes to injure both purposed are 
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served, there are advantages to separating them. 11 It was also thought 

that locally based evaluation would "seem to be more effective and to have 

more local credibility ."^ 

The third evaluation level -was the least developed but most 
ambitious. Level III was intended to aid, pull together, ,and sum- 
marize the other two levels.' It was intendeds to 
look ,! across J projects ... to provide technical/professional^^ 
assistance and cross-fertilisation; to provide informed 'criticism 
of the methods and conclusions from each level [of the evaluation], to 
derive and implement evaluation approaches and models that may be 
used to assess the merit of comprehensive experimentation, and to 

identify and formulate evaluation problems which may be turned over 

17 

to basic research programs for investigation." ESP Washington 
thought that the major "users" of sUch an evaluation would be federal 
policy makers, who would be able ta replicate- elsewhere the successful 
programs and evaluation methods identified by it. 

r 

* 

Thusf^all th?fee evaluation levels rested on certain assumptions 



^ evels rested on certain assumptions 

aboitt the information each level would provide and the ways in which 

it would actually assist practice. There W£#\no clarity about how the 

evaluation was, to be implemented, but two central notions were plain 

enough: it was to produce new knowledge on the local change 

proc'ess; aud, through application of that new knowledge in federal 

and local practice, evaluation would facilitate local reform* 

* * - 

Yet the federal assumption that comprehensive change could be 

evaluated presented an .iwaediate obstacle to plans toy the evaluation. 
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Since the notion of comprehensive change was itself very unclear\ 
those asked to assess it did not know what to do. This provoked ar- 
guments over the definition of comprehensiveness and how to evaluate 
it , .frustration at the enormity of the evaluation task itself, and in- 
stability in. the evaluation staff. In the end, this innovative evaluation 

produced rather weak results. It gradually became more traditional, its 
methods more familiar, and its aims less ambitious, 
Struggles with Comprehensive Change and Holistic Evaluatio n 

To improve upon the limited success of p^st social science eval- 
uations, the ESP evaluation sought to understand better !, how the 

process of change takes place, 11 and to provide practitioners with 
knowledge about program outcomes,. One r of its major components was 

a novel anthropological/ethnographic approach that wouLd allow on- 
site researchers^ to study and report on the local change process as 
it occurred. This docuJ^itation of the "natural history 11 of the , 

project was to deal with - 

j 

. , . the project as a totality. The interrelationship 
among the components of the project andr J:he nature of # 
the forcing [sic] functions, which promote or inhibit* 
successful project development, are to be identified 
\ and assessed with respect to relative influence or 
impact. 19 

It was hoped that such a method of study would provide social science 
knowledge more useful than that generated b^ the quantitative focus 
of previous outcome evaluations, because it wouLd include all aspects 
of local change and not be narrowly circumscribed by quantitative methods . 
The expected "holistic" information about the local change process could 
then be used by federal ESP program managers, as well as by others who 
wanted to reform local practice, 
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While a qualitative approach was emphasized, statistical 
• studies also were required. The federal plan was "to include f 

but not be limited to student achievement, attitudes, staff performance 

20 

and other more traditional categories of evaluation. 11 But as the . 
federal director of the evaluation pointed out, the aim was ft to 
learn how the process of change $ook place"; 'he was personally con- 
vinced that there was little in conventional quantitative evaluation 
that was worth. doing. This more traditional criterion in the Request 
for Proposal (RFP) was apparently due to pressure from the HEW office 
of the Assistant Secretary for Planning and Evaluation, which, in the 

vien of ESP staffers, "did not think an evaluation without pre- and 

21 m 

post-testing data was an evaluation." 

The burden of ^devising the non-traditional approach to evalua- / 

tion fell on the shoulders of the level II contractors. The 

RFP articulated this assumption: ". . . in the course of the ESP, 

4 

many needs will be identified for the development of approaches to 

evaluating educational settings. ... As the new approaches and 0 

topis are developed, it is expected that the^ will be tried in the 

22 

field by the evaluation and documentation contractors . " 

Once federal ambitions for ESP evaluation were translated into 

an RFP, potTential level II contractors came forward with a hopeful 
/ . * 

and accepting reaction. They expanded the already grandiose federal 

•> 

plans for ESP. Responses typically expressed awareness of the 

failure of past evaluations, and wade vague promises- to use social 

science methods to produce aore^fruitful knowledge on the local 

change process. One level II document phrased this in the following » 
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way: "Evaluative researchers have relied primarily on mathematical 

models (the traditional research models of sociology and psychology) 

ta j^dg^ith* success or failure of ^educational programs, . . . This ' y 

[mathematical] approach to evaluation, which is too narrow in focus 

and has limited evaluators 1 ability to do more than speculate about 

program results^ led to the general state of dissatisfaction of those 

being evaluate^. . . In its place, this contractor promised a 

more quali tattvely oriented evaluation. One proposal even offered 

a line of inquiry so novel that it would help create a new profession: 

"In recent years, evaluation has become a tdkhnical specialty with 

associa^l methodologies, canons .of proceduje, fairly abstract bodies 

of knowledge and theory and accumulated experience, a set of more or 

less renowned practitioners and a vested interest in developing all 

of these further. Our own evaluation efforts are expected^to make 

a contribution to this rapidly growing field which threatens or 

promises to 'provide grounds for a new and distinct profession." 24 

The evaluation, then, was to build social science knowl-edge on 

educational change, But in the absence of a usable definition of that phenomenon, 

level II contractors quickly came to see the ESP program aim—comprehensive 

change-as an impossible target for evaluation. As one level II con- 

tractor explained, "You must recognize from the outset that this 

situation made it difficult to do research in a way which would qual- * 

25 

ify as research to the outside/ 1 Another contractor, explaining 

the low^level of productivity in his evaluation, said that the notion 

of comprehensive change was Qnly "a metaphor 11 with "no reality" in ^ 

the. world of social science research. A former member of a level 
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' II contractor 1 ^ staff explained that the original RFP was "awfully 
vague 11 and he "did not know how much time [his firm] .spent debating 
comprehensive changed 1 - Another" member'oT the s&me firm added that 

his firm later* "pushed NIE to define it."~ A staff member of another 

<~- 

contractor referred to* the difficulty by saying that M the lack of a 

defined treatment really came home to roost because it was difficult 

- 29 
to know what to measure in terms of outcomes,' 1 And to make matters 

worse, many level II evaluators felt unable to share their doubt 

about comprehensive change with their client. 

The samer prob lem arose in the "ethnographic" component of the 

level II evaluation. One level II staff member called most of his 

own work "essentially meaningless in the ^context of the program.""* 1 

He explained that what ethnographic reporting was done was hard to 

relate to documenting "comprehensive change," since that notion was 

so ill-defined. The director of a level II evaluation touched on the 

problems of communication with federal program people over this issue 



< when he said, "We talked a lot with NIE about the trouble of trying # 

,,33 



f 
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to use NIE ! s five fa.cet? and that we would have preference for 



social science categories, but that was another area of negotiation. 
This comment went to the root of the problem: the contractor found 
the federal ESP notions problematic, but the federal staff did not. 

*One resulting problem was a iack of federal criteria and coherent 

/ 

guidance for ESP evaluation. Federal managers ^wanted the evaluation 

♦ 

to provide a "holistic 11 view of local change efforts, yet they didn't 
know what this mean£. They knew tfnly what they didn't want, and 7 
sensed to think that by naming an alternative they had created one. 
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But to call evaluation holistic provides neither criteria for judging 
what^hdt is or is not, nor guidance for how to do research. Given this 
lack of guidance, and given that not all aspects of local change 
efforts could be studied, level il contractors had a problem: which 
aspects to evaluate, and how? .There were no consistent answers for 
any one contractor, nor consistent answers among them. Indeed, the 
-problem was compounded by the federal ESP staffs interest in explor- 
ing various features of comprehensive change. . These features changed 
from time to time and from^monitor to monitor, and this created more 
uncertainty for the level II staffs. Level II contractors were con- 
tinually trying to do too much at once, or too little, or the wrong 
thing, or some ^combination of these. The evaluation was never given a 
central fpcus, and as a consequence, work floundered. 

Numerous conflicts and dtfbates arose, over what aspects of 

local change efforts might be appropriate for study, ajid with what 
•social science tools. But once level II contractors de- 

cided these matters in some way, they placed limits on the work, and # 
then federal ESP staff could see evaluation efforts that 

were incomplete and too specific-- for there were always more aspects 
of local change to be studied. Level II contractors, with the limited 
tools of social science, and limited time, simply could not provide 
the ,4 hol ism"-- that fantasized complete picture of local change—that the 
federal staff dreamed of. 

Thfe ensuing demands placed upon level II contractors contributed . 
to instability, confusion, and even capriciousness in the evaluation. \ 
The rate of turnover of project directors was high: six of the seven 
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original directors resigned during the course of ESP. Turnover rates 
for" other level II staff were also high.'* 4 The lack of criteria for 
evaluation also' allowed federal monitors to change their theoretical 
and methodological priorities continually. As one level II staff 
member explained, "ESP 'had a history of flip- flopping 180°, and as 



.the project officers changed, so did the direction given to the eval- 
uations . tf,>5 Another contractor mentioned the same problem: "Each/ 

* / 
* ! 

federal project monitor had their [sic] own idea of what the rural' 

school evaluation /should look like. Consequently, they kept asking 

for changes that w\>uld suit their impressions. But often their/ 

changes had not been agreed upon earlier . . . and since they qbuld 

not easily be defined to third party researchers, we stuck to (^ur 
36 

design/' Other contractors, not so self-assured, often chatiged 
their design and data collection procedures in response to the demands 
of a federal monitor. 

r 

This situation made it difficult for contractors to formulate evalu- 
ation plans that were acceptable to federal reformers* Contractors of^en 
had to rewrite evaluation proposals — in one case, a proposal had to be 

rewritten four times in one and a half years; it was never approved and 

- * 

* • 37 

the contract was' eventually put up for rebiddipg. The difficulties • 

continued throughout the early part of the project. Several level II 

contractors were never able to produce an acceptable^plan, even years 

after the evaluation was under/way. In fact, two of the seven evaluation 

contracts were put up for re Ridding. ' 

* / * 

In these circumstances there was constant friction between federal 
ESP staff and level II contractors. The ESP staff saw a grea£ discrepancy 
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between what they wanted gnd what contractors produced, and they saw con- 
tractors as incompetent, mercenary, or both. One monitor explained rather 
simplistically that the contractors were motivated by greed and had little 



"core" interest in the possible fruits of such research. 
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Another common 



view was that ESP had given "the wrong people contracts," people who 

\ 30 
"could not deal with the freedom built int& the design." While the 

i r 

latter comment has some basis, it cannot explain the often disastrously 
poor quality of the evaluation reports, the inconsistent federal manage- 
ment of the evaluation, or the connection between the two. 

These problems *did nothing-to improve the evaluation. In several 
cases, level II contractors, facing the difficulties of trying to formu- 
late an acceptable approach teethe evaluation, decided that comprehensive 
change could not be evaluated. They gradually abandoned the non-traditional 
and qualitative for the^more familiar quantitative social science methods* 
As one level II staffer explained, "NIE seemed so muddled about certain 
^things, we felt the only thing we could do was try and publish so that the 
social science community would know what we were doing." 4 ** A^6ther said it 

more simply: he and his colleagues just "did their own thing and nobody knew 

41 • « 

what the right thing was." Given training and professional values, "their 

own thing" often tunied out to; be traditional social science. 

Thus in many level II evaluation reports, a major element of 
the federal evaluation plan—documentation of the change process 
through ethnographic/anthropological methods— yielded, to t^e^nfamous tra- 
ditional quantitative approach. In one level II urban evaluation, 
studies and reports were' based on faailiar social science citfegoriesl 
''Student Achievement," "Attitude Instrument Study,"* "Language \Domi- 
nance Assessment," "Socio-Cultural Bias of California Test of Basic 
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Skills, 11 "Self -Concept Pilot Study." Another urban site* contractor 

submitted reports on other familiar subjects- -"Survey of Parents in 

the Jefferson School District," a "Study of Curricular Decision- • 

* * 

'Making in the Jefferson Experimental Schools Program/' "The Status 
of Individualized Instruction Strategies on the Elementary Level K-6 
in the Jefferson School District," and "The Effects of Transiency 
and Social Integration Upon Academic Achievement . 1,43 

. In ESPs closing years, this shift in emphasis from non-traditional 
and qualitative to more conventional and quantitative methods found a 
parallel at the federal level. Federal ESP staff became aware that 
they were not receiving the novel information and insights expected" 
from the evaluation, that in fact they were receiving a very small 
return on their investment altogether. With the prospect of having 
little to show for an innovative and costly evaluation, they 
tended to emphasize a more traditional quantitative approach, which 
at least might yield some credible results. 44 

% This search for credibility at the federal level was intensified 

by the transfer of ESP, from the Office of Education to the newly 

established and research-oriented NIE, one and a half years after 
45 

ESP's start. One program officer explained the transfer as a 
declaration of "open season on ESP," adding" that nobody in ESP, though', 
really knew how "to ftlay the research game." She was referring to the ESP 
staff 'slack of social science knowledge required \- to defend, the 

evaluation, — She-also argued- that "ESP .was like boiling an' egg-^cdu 

cannot keep pulling it out of- the water every five seconds to see 
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how it is doing, ,f Her views were to no avail. 46 Another federal 

staff 'member also noted that pressure was growQjg : "People . . 

[at JfJ£] felt that the only way you could justify anything is from 

hard data or . . . they wouLd not think it was a responsible evalua 7 
1 . 47 

^tio^ 11 Almost by accident, then, the transfer created a point of coi 
gence for level II contractors and ESP Washington staffers that had 
not previously existed. In one case, a level* II contractor suggested, 
'artd had approved b]^ ESP Washington, a major reorganization of the 
study three years after its-conception, which cut back on the ethno- 1 
graphic research and offered a more rigorous traditioiJlPstudy of 

AO 

pupil change (in ESP and nan-ESP districts). This new orientati^ 



toward a quantitative rather 'than a qualitative study was a" substai^) 
> tial departure from the initial intentions of the federal ESP staff/ 

but it did respond to new Washington ide.as about evaluation. 
• * ... » ,^l x 

Despite feSeral efforts to improve the evaluatfflft f Vts products 

remained undis,tinguisfifed. * The ethnographic work was typically descript 

49 

-and had little analytical content. For the most part, the .level * 

triedi^tq; record and- -"cfociroent" what occurred in the different 
parts of a school system. The reports seemj^|4^oous arbitrarily 
on certain/events ot aspects of a district, describing activities in 
*€Us schools; in a somewhat journalistic fashion, This was not the 
disciplined analysis arfd observation expected iiJ research; neithe* 
was it the. focused picture of events fdund* ip good reporting. It 
was> instead, often vague, diffuse, »at\d undisciplined bj any analytic 
fraflffe| It offered fewj&jisights into^comprehensive or any othW 
local change processes-. * ' 
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The lai&r^ and more traditional .quantitative work suffered from 
other limitations. Understandably, it lacked a definition of the 
experimental school "treatment" studied; the treatment was unspeci- ' 
fied, or only vaguely indicated, in mo^ievel H reports. In one 
instance, an evaluation employed 'a series of comparison studies 
6f ESP treatment and non-E^P Control students, yet there was no dis- 
cussion of how, or on what basis, the treatment and control groups 
were determined.^ 0 Another contractor was simply unable to define a 
treatment, and J?ft the issue unclear in proposals and reports. 51 In 
yet another sase, a seemingly measurable treatment was 

chosen, but it^had no relationship to the federal program notion: 
the level II contractor, after various efforts to dcfin6 a feasible 
ES treatment, decided on the definition of "dollars delivered to 
schools," 52 Other reports explained simply that the ESP local project 
"stajff "designated certain grades" as experiencing ES treatment; 53 jbx* 
that ES students represented a sample "not in the sense that they were 
systematical!^ chosen but that they are a subset of the totaj number # 

of students who were available for retesting or who were actually 

54 t \ 

tested." In one study, control schools were selected on the basis 

of gepgraphic proximity to ES treatment schools and willingness to 

. . * 55 
participate in the study. 

Even when student "populations were identified (by whatever 

f 

^scientific means), they characteristically, became unstable, or 
incomplete, in what they were to represent. In one, study, 263 stu- 
dents were tested out of an initial population of 862 "ES students*" 

♦ - 

* * 
The report also implied that there was no assurance that these 263 
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students were* part of the initial ES population, owing to poor — * 
records on student enrollment, absenteeism, turnover, and dropouts..;^ 

i 

Another report explained that the total number of students tested in 

the treatment a#d control groups ranged from 50% to 85V of their re- 

spective total populations identified each year over a four-year ~ 

period. There was no indication that the students in either group 

were part of the original fir^t-year sample. 5 ^ 

4 

< 

.Some level II contractors simply did not produce reports, in 

part because of their own awareness of such severe shortcomings 
t 

as those cited above. Rather than produce work of poor quafityy 

they chose to submit no reports, or purely theoretical reports on 
social science research issues only vaguely related to ESP, A memo 
from an NIE evaluation monitor indicates this tendency in a level II 
contractor's work two years into the evaluation'. It reads: "As was 
feated . . . the conceptual model for organizational studies and « 
analysis continued to be very abstract and failed to deal wit}! the 
concrete operational ization of the research dimension. At this point 

the model itself as well as * . % [the] ability to implement it remains 

58 K * * 

unknown. " The work of the level II contractors was summed up by a potential 

contractor bidding for the level I LI evaluation. The proposal read: 

• No mechanism presently operates for- returning to the 
research community or policy makers anything more 
than an expensive set of case studies on local 
history. 59 t- 
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In- summary, the attempt to produce new knowledge on local change 

was a 'fiasco. One reason for .this was 4 naivet/ about social science 

* * • 

methods on the part of the federal ESP staff. Another was the prdblem 

i 

of evaluating coraprehe/isivfe change, or of carrying out a •"holis'tic" 
evaluation. In addition,, th§re was'. weak federal management of and 
guidance for the evaluation, * due in good measure to the combination 
of grandiose and naive thinking that ftarked the program's management. 
ESP f s major promise—that change can be comprehensive, and that it can 
be evaluated, holistically with social science methods — remained unfull- 
filled. The evaluation reports produced by level II contractors were, 
with £ few exceptions, cpmpletely undistinguished, and many were an 
embarrassment. 

Management and Knowledge ] 

This social science knowledge also was' supposed to improve 
federal and local management. The ESP staff reasoned "that past eval- 
uations had failed in this regard partly because the sociai science 
knowledge had been of the wrong sort, and partly^because* evaluations* 
gave rise to political conflicts between federal and local prac- 
titioners. They -proposed two remedies. First, the; evaluation was 
to produce knowledge ( reLevanfcxto the needs m of prac- 

titioners~to "provide the practitioner with adequate information to 
allow him to j&dge whether or not ch^g^/ in his practice are war- 
ranted." 60 And second, to -reduce conflict and suspicion about the 
nature of the soci'al science knowledge produced, evaluation was to 
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take place at three levels, presumably geared to separate local and 
federal program management needs. 

The attempt to design an evaluation <hat would facilitate program 
management foundered, however, on differences between federal ideas and 

local practice. Federal reformers believed that the new know- 
ledge emerging from the different evaluation levels could be helpful 
to federal and local practice, but local practitioners did not 

respond as expected. They had knowledge, ordinary and professional, 
that seemed adequate, and saw little use for the research. 
Moreover, they saw federal efforts to generate social science knowledge 
from the evaluation as an instrument of political oversight for 
their federal managers. - This created tension between federal and 
local practitioners and set the stage for much conflict. Thus, the evalua- 
tion affected management of the program, but it hindered rather than helped, 
The Level II Evaluation 

Local administrators not only showed little interest in or use 
for the social science findings produced by level II contractors; 49 
typically, they saw the level II evaluation as an independent research 
Tvroject unrelated to local operations. Often they regarded . 

it as a ryuisance, since the evaluation required much local time and 
cooperation. .Thus, in sites where the v social science findings of level 
II contractors were made available, there is little evidence that they 
wer^ helpful |to local management concerns and interests; One adminis- 
trator explained that he looked at the level II reports "belatedly" - ' 
because "they got into gossip rather than the real issue." He added 
that he had his own inside information on what was going on in the * 
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s^pfT&ol districts and did not need to be informed about the n popular 

I 61 ' 

image ' of how things were. Social science knowledge did not 

correspond ^to what local participants believed was going on. 

The* low utility of the level II evaluation for local administra 

tors was reflected in lackadaisical attitudes toward 

facilitating data collection. One superintendent, who kept getting 

letters from the level II contractor "asking us to help get the ques 

62 

tionnaftes filled," said that he responded only out of deference t 

» 

Washington. Local administrators had their own ways of gathering 
knowdedge through -local .staff , and th$y saw little to be, gained t 
from impersonal questionnaires. Furthermore, their own decision 
making depended upon considerations ancTf actors that were peculiar 
to their particular situation, and seemed worlds apart from more 
universalistic social science methods of gaining 'knowledge. 



Teachers had a similar reaction* They saw level II requests 
for information as irrelevant, of little use to classroom manage- 
ment. One elementary teacher complained about questionnaire's 
asking, f<?r example, how much time was spent on discipline in class, 

or how many students in the class smoked. She couldn't see how the 

s 

answers would in any way help her in the classroom. As a result, 
she filled out the first one but "otherwise I threw them in the 

fireplace." Another teacher said, "I don*t know much about the 

64 * . * - 

evaluation; it bo^es me," Teachers rarely read the reports issuing 

from the evaluations, and when they did, the response was the same. 

One teacher noted, H I saw the» tyst I did not understand thea and I 
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did not know how to read them," 65 The knowledge and skill required 
for teaching practice overlapped little with the social science 
knowledge of the evalua£ions/ nor did the second illuminate the first. 

Local administrators thus often found it difficult to facilitate 

/ 

level II data collection, even when they wished to. Teachers, prin- 
cipals, and students were already subject to testing and other 
* research efforts, either from other evaluations or from local 
testing programs, or both. Many resented and resisted 
the ESP program. Their poor response to level II re- 
quests was undejstandable, but it, impeded t fie work of local admini- - 
strators, who often were caught between the demands of Washington 
and level II Contractors for data, and their own staff's hostility 
or indifference to another evaluation.. 

But if most local practitioners had little use for social science 
knowledge^ the evaluation was not irrelevant. Many of them saw 
the level II evaluation- as a way for ESP Washington to pass judgment 

# 

on and manage local program activities. This produced considerable 
tension between federal and local administrators^ and contributed 
to more than a few management problems, Locals saw level II con- 
tractors as federal agents, 66 One local project director said that 

the level II evaluation was part of the government's "experiment in 

* ' 67 

schools, " and felt "we were guinea pigs." A local ESP staff member 

said it was his "impression- that the level II people were almost 

like a fiscal agency representing Washington at the si£§ t " 68 A level 

II project director explained that the local sites he was evaluating 

had difficulty even telling "the difference between NIB and the CIA, 

^' ■ $2—: ' 
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never mind the difference between my own organi/ation and the federal 
69 

government." This view of the level II evaluator as a federal 
agent was pointedly illustrated in a rural site whose community and 
school staff objected to government-sponsored evaluation because 
they suspected that the level II questionnaires on the morals and 
attitudes of their children would eventually be used by the govern- 

4 

^ment for measuring/antisocial behavior--"another version of Orwell's 

^ 1984 , tf7Q Local practitioners typically saw the level II evaluation 

in adversarial terms, as serving federal purposes. 

As local sites were often unsure of what ESP Washington want.ed 

their programs to be, they were particularly concerned about the 

effect of level II contractors 1 reports on their funding and their 

relationship with Washington. As one local person described her 

.sitels reaction, they "feared the evaluation would ultimately limit 

71 

-What they could do." v The director in an urban site explained that, 
particularly at the beginning of the project, there was a great deal 
of hostility toward contractors who "apparently had not even read our* 
proposal before they submitted their plan for evaluation--it was 

* 7? 

hard to understand how they could make judgments about the project. 11 
Local practitioners typically felt that the level II evaluation re- 
ports were out of kilter with their own knowledge of the program, ■ 
and criticized contractors' interpretation of program objectives and 
implementation* 

This divergence in views fed local suspicions about the evalu- 
ation, produced extended arguments over the approach and findings of 
contractors-, and filled political conflicts. .In one project, the 
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local staff and community went so far as to demand the right to 

review tests before they were given, because they felt that the p^oj- 

73 

ect's reaL*goals were not being measured. In another site, the 
project director said that the level II evaluators never quite under- 
stood what was going on in the district, and that their reports 
"were so subjective that they only made me angry--they reported 

things that never happened and took things that were entirely out 
74 

of context. M In a third site, anti-establishment sentiment in the 

r 

community coincided with the program staff's concern over a "govern- 
ment-sponsored* 1 evaluation and virtually precluded agreement on 

what, if any, evaluations should take place. ThV level II director 

/ 

f 75 
reported that this attitude "immobilized any research," 

\ 

k 

'The federal ESP staff were in a most difficult situation, for 
they were hard pressed to judge the validity of local criticisms. 
While they were skeptical about both the evaluators and the conduct 
of local projects, they could rarely spend enough time on site to 
form their own conclusions. In fact, they were busier trying to re- * , 
solve conflicts between level II contractors and local schooT people 
than absorbing contractors 1 information on the programs. Typically, 
they tried to walk a narrow line between local complaints and criti- * 
cisms, and level II responses and counter-accusations. The evaluation 
^component of ESP thus hardly contributed constructively to federal 
program management. According to the federal design, social science 
knowledge gained from the evaluation would aid in program decision 
making and management, but the federal ESP staff certainly never relied 
on the evaluation, 9r other social science knowledge, in their own 
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program management.. Indeed, they questioned the credibility of the 
level II contractors' social science knowledge, which was often incon- 
sistent with what they kffew, and relied "instead on their own professional 
and ordinary knowledge in managing ESP. One federal staff member- ex- 
plained that J'the evaluation unnecessarily complicated" the program 

effort and "just did not make sense pr a dent in the day-to-day opera- 

,,76 
tions." 

Ironically, the conflict also made level II contractors reluc- 
tant to supply Washington with what findings they did have on local 
-programs, for they feared that federal managers would use such informa- 
tion in their management of local projects, and that this would further 
strain the evaluators' relations with iocal staff. The 
contractors recognized that a workable relationship with their sites 
was essential, and so argued that their major purpose was to conduct- 
research "in the school site which would take place over a five-year 

• , * 77 

period, , . . not to give N1E information about the project," The 

' f \ 

reasons for this view were made explicit by one level II director; # 

"We became very leery of our ability of keep our commitment to the 

* 78 

sites if we gave much information to Washington." < 

The reaction of ESP Washington to the situation was deepening 
concern about whether the evaluation work was in fact being 'done. 
In some cases it was not, as a result' of difficulties mentioned 
earlier that level II contractors did not care to share with ESP 
Washington. In other cases, work completed by contractors was sertt 
on to ESP Washington only after it hadMaecome irrelevant to local 
events, and was of little value to federal monitors. The level 
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II contractors were in a no-win situation: if they passed pejorative 
information on to the federal ESP staff, local sites might prevent 

w 

further data collection and make it impossible for them to^fulfill 
their contracts; yet if they, did not share at least some of the in- 
formation, the federal staff might suspect that they were not ful- 
filling their contracts. The decision to withhold information so as 
to maintain an amicable relationship with local sites thus created a 
new area of conflict with ESP Washington, and a new management problem. 
The national staff • responded by pressuring contractors to divulge infor- 
mat ion on 3local sites. 

^These problems arose from evaluation/ and were not resolved by it. 

They differed little from those found in other evaluations. They 

i 

are perhaps best illustrated by the events surrounding the federal 
desire to review. level II ethnographic 

studies, especially those of rural sites. The ethnographers, 
who were suspect in local sites to begin with, were particularly con- 
cerned that if information ^ey gathered was seen by ESP Washington, # 

it might be used against local sites, and that their relations with - 

79 [ / 

the communities would be jeopardized. Consequently, individual ethno- 
graphers even hesitated to give their information to their own (level 

80 

II) employers, for fear it mi-ght be passed on to ESP Washington* 
ESP Washington, on the other hand, wanted to be sure that the work 
was being d&ne, and was of reasonable quality. They feared that the 
ethnographers had "gone native 11 and were holding back information. 
To resolve this problem, an elaborate review system was eventually 
worked out: the work of the ethnographers would be reviewed only • 
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^by people outside ESP Washington, and the reviews/ would be an assesS- 
ftent only of quality, rather than of substance. / This system 
a^ded yet another management burden, and cost, to the federal ESP staff 
It; caused extended delays in ESP Washington f s /examination of level II 
contractors 1 work, and of course rendered the information utterly use- 



less for 



ESP management, 



... parallel conflict ajose in connection with the plan that level 

II evaluation should provide information useful to local sites. The 



orig 



°f 

inaj federal notion w^s that this would minimize political 



protklem^, as local &ites : : woyld reeeiye information from the same 
source /as the federal ESP office. Instead, level II contractors 



agail^i ^hesitated to transmit information to local sites for fear of 
damaging the relationship, and this increased local resentment of * 
the evaluation, As/ one superintendent explained, the failure of 
level II contractors to feed information back to their sites was, 
in his mind, "the height .of foolishness,", Me found it particularly* 
infuriating because they "burned the territory once or twice a year 

asking the community questions and then not giving answers fsic] to 

83 4 

us. ,M Another projectldirector reported that he and his staff did 

not have "the slightest idea of* what they are doing. The only thing 
we got out of |the level II contractor] was the Iowa (Test of Basic 
Skills] Printout." He added that he was^old he would "never see 
any of the information, so don<t ask for it." 84 The reticence of 
level II contractors did not relieve local administrators* suspicions 
that the evaluation was inaccurate and/misleading, ^ - 

Some level II contractors dealt with this problem by issuing two 
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separate sets of reports. On^ leveV II project director explained that 
he sent local schools* special reports, containing little information, 
because he feared that^if the site persorfnel saw the real reports they 
might cut off further data collection. So in place of the real 

reports he created and sent an "innocuous report or two to keep them 

_ « 

85 

happy. 11 Another level II director did the same, simply giving sites 

son|e general material for M a political purpose"--so that they would 

not contest the information or want a part in the data collection and 

analysis. From the level II contractors' point of view this was critical, 

"as a strained relationship to the local sites would completely blow 
x • " 86 

\ ''our -effort to do a summative evaluation/ 1 The crazy politics of the 

\ .! 

evaluation thus meant that information given to local districts could not 
• be useful to them,, for if it was it would be unhelpful to the evaluators. 
The information passed on to the districts^as intentionally made use- * 
less, so that the evaluation could proceed'* In practice, the idea that 
social science knowledge would provide a management tool for local dis- 
tricts resulted in the deliberate creation of social science fictions, in 
order that the relationships between evaluators and locals could be made 
manageable. 

The Level I Evaluation • 

The level I evaluation shared many of these problems* It was 
tc have been locally operated, to help districts improve 
project management and reduce political conflict. .But local 
sites, particularly rural ones, typically did not understand the notion of 
creating a forma U system of social science knowledge. As one rural school 
director explained, he knew "nothing, absolutely nothing" about the level 
I evaluation except that "ESP in Washington ^aid we had to have it." He 

, B8 , ~\ 



added that "the people in the project had no idea what it was for. * They 

were practitioners. .Nobody did research .... They were normal people 

and could not be expected to do a formative evaluation." 87 Similar * 

views were expressed even by urban sites, which were more familiar with 

evaluation. One project director said, "Quite frankly, I did not know 

what [level I evaluation] meant, but we tried to- make it part of the 
88 

project." 

Other local managers were passive about -or resistant to level I 
« 

©valuations because they saw little value in the knowledge to be 

% 89 ' 

provided. Thus, they recruited local people as evaluators who had 

90 

little experience in the area of evaluation. Federal managers 
were not pleased, elTher with such selections or with the work 

produced, and even tried to thelp find better- 

91 1 * 

qualified people. They saw the level I evaluation as a way of 

helping local sites to make program-related management decisions,} 

but few locals saw any use for it. The harder the federal 

staff pressed the locals to take the evaluation seriously, and the # 

a 

more actively they tried to set it up, the more local sites suspected 
that it was just another way for the "feds" to evaluate them.. The 
federal staff could not believe that local practitioners had little 
or no need for social science knowledge, even if it was their very own. ■ 
9 In some sites, no level I evaluation worth mentioning took place. In v 
those local sites where some sort of evaluation took place, it often ^ 
caused additional problems . Teachers and other local staff were restrained, 
apathetic, or hostile. One level I evaluator, for example, talked about 
how* difficult it was for local staff to "write their objectives." He ^ ; 
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explained that "this was not ^typically the conCern of teachers and educa- 
tors,, that \s, to systematical l^formulate their process, because they are 

o 

used to operating intuitively and by the seat of their pants." The skill 
and knowledge that worked in the t^l ass room, he believed, was not that of 
social" science . .And there was the further difficulty that, since 
gathering social science knowledge was described at the federal level 

as an* evaluation, "evaluating wili always serve as a threat to tea- 

• '92 " r 

chers." His reaction was not unique. Local teachers were c|lsturbed 

over the amount of data collection, and level I evaluators found it 

difficult to obtain information on the program because of the resulting 

local staff resistance.' 

A 

As a consequence the reports that did emanal^? from level I were 
hardly used for the local* management of ESP. Typically, only local 
administrators were even aware of the Imports. As one level I ^valuator 

said, "Nev^r has the evaluation been taken seriously in the central 

93 * ' ' ' 

office." Other central administrators did use the evaluation, but 



generally to confirm what tiad already bee^decided or known informallf. 
One superintendent explained that ifr often "helped to confirm our 
intuitions and feelings." Most, however, found it a bother. 



tin the ,whole;„ then, level I evaluation was not a great success.^ 
In some £a^s* it seemed so irrelevant that, it was never even iftple- |^ 
mented. In others, it added another area of conflict between fed- 
eral and local managers, and contributed to .local administrators 1 P 
internal management problems because of local staff resistance. At 
the core of these problems was the erroneous^Qprption that Ipcal 



administrators would use social science knowledge as an aid jto man- 
agement, especially if the evaluation was locally based. Instead, 
local >practitioners^had their own sources of knowledge, and had littltf 
us for level I evaluation. Typically they saw it merely as another 

4 

95 

way for federal staff to monitor local efforts. 
Conclusion 

The ESP evaluation scheme was intended to provide new social science 
knowledge on the "local change process, and to facilitate program 
management. But this scheme encountered the same problems as did other 
federal evaluations. The social science knowledge gained ?rOm the evalu- 
ation was of poor quality, and it contributed little to practice at any 
level *of education. Its only appreciable effect was to ilterease conflict 
within the ESP program, and to raise management problems all along the 4 
line. Indeed, in this sense it created a new area of practice: managing 
unusable but potentially problematic social science. This has become a 
large element in t,he management of evaluation in other federal programs, 
including Title I and PL 94-142. 0 * 

Ironically, though, the problems that beset the ESP evaluation did • 
not stem from the poor quality of the knowledge obtained. * We suspect' 
that the opposite may be true: knowledge of higfier quality might have 

■ 

intensified rather than diminished political conflict. In t^e local * 
world of practice, social scienie *knowledge--however good it may be, and 
however useful for other purposes—will not replace o*her trusted, ordi-^ 
nary, ways of knowing. But while the social science knowledge produced 

by* evaluation is of little use in program management, practitioners 

• » » >, 

fear it will be used by others to limit their freedgm # of action. The j 

)' .. . i 
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more effective 1 the information, thep^ probably the greater the threat. 
If the ESP studies had had greater credibility, they would thus have 
received even more attention. This would have increased local ^suspicion 
and resistance, political tensions between^ outside evaluators and local ' 
program operators , and federal defensiveness . Conflict would likely 
have been etxacerbatJed . This raises questions about the long-standing 
assumption of federal policy that social science knowledge relevant to 
a program will be useful to program management. 
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.Doyle, Wayne, et al. Final Report of the Northwest Educational Regional 
Laboratory, The, BirtK, Nurturance "and Transformation of an Educa- 
tional RefQrnT Portland,* Oregon: Northwest Regional Educational 
Laboratory, August 1976, p. 5^- 

24 * ' / 

Report submitted by a division of Scientific Analysis Corporation to the 

ESP office in the National Institute of Education. A Conceptual 

Overview of the Evaluation and Documentation Effort . Berkeley, CA. : 

Scientific Analysis Corporation, December 20., 1972, pp. 11-12. 

2S 

Interview^ith staff member of a level II contractor, Abt Associates, 
Cambridge, Massachusetts, March 29, r 1977. 

26 < ; 

Interview with director of a level II evaluation,. Scientific Analysis 

Corporation, Berkeley,' California, August 3,1, 1977. 

27 * 

Interview, with staff member of a level II contractor ,< Northwest Educational 

Regional^aboratory, Portland* Oregon, December 2, 1976. 
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28 

Interview with staff member of' a level II contractor, Northwest Educational 
Regional Laboratory, Portland, > Oregon, April 14, 1977. 

Interview with staff memher of a level If contractor, »\bt Associates, 
Cambridge, Massachusetts , March 24, 1977. 



^Interview with director of a level II evaluation, Northwest Educational 
Regional Laboratory, Portland, Cfregon, December 2 ; 1976. 

31 » 

w Interview with staff member of a level II contractor, Northwest Educational 
Regional^Laboyatory, Portland, Oregon, December 2* 1976.* 

3^ 

"See beginning section on the federal definition of comprehensive change. 

This definition listed five facets (i.e., curriculum, organization, 
instruction, administration and community) which comprise 

t w comprehensive change. 

■* 

^Interview with director of a level II evaluation, Abt Associates, Cambridge, 
Massachusetts, August 16, 1977. 

^Interview with staff member of a level II contractor, Northwest Educational 
Regional Laboratory', Portland, Oregon, December 2, 1976. 

33' * 

Interview with staff member of a level /II contractor, Aries Corporation, 

^Minneapolis, Minnesota, June^, 1977. 

36 Interview with staff member of a level II contractor, Abt Associates, 
Cambridge, Massachusetts, March,24, 1977. 

37 

Telephone interview with director bf a level II evaluation, Human 

Interaction Research Institute, a subcontractor of the Scientific 
Analysis Corporation, Berkeley, California, August 31, 1977. 

38 

Interview with senior federal ESP staff member, November 17, 1976. 

/ >_ 

39 

Interview with federal ESP director of evaluation, July,27,' 1977. 
/ m 

4 °Interview with staff member of a level IT contractor, Abt Associates, 
• - . Cambridge, Massachusetts, March 24, 1977.- 

I - , 

41 Interview with staff member of a level II contractor, Northwest Educational 
Regional Laboratory, Portland, Oregon, DeCemtier 2, 1976. 
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42 n . . 

Reports submitted or in preparation for submission to the ESP office in 
the National Institute of Education by level II contractor, D. C. 
Development Associates of San Antonio, Texas. 

*** * ,* 

43 n 

Reports submitted to the ESP office in the National Institute of Education 
by level II contractor, Northwest Educational Regional Laboratory, 
Portland, Oregon. 

44 _ 

Interview with director of level II evaluation, Scientific Analysis 

Corporation, Berkeley, California, May 12, 1977; interview with 
director of level II evaluation, Aries Corporation, Minneapolis, 
Minnesota, June 3, 1977. 

45 V* 

Interview with federal ESP staff member, December 3, 1976. 
* / 

interview with senior federal ESP staff member, November 17, 1976. 
47 v 

<>otes on a telephone conversation with federal ESP staff member <?n 
September 19, 1977. * 

48. 

' interview with director af a level II evaluation, Abt Associates,- Cambridge, 
Massachusetts, August 16, 1977. 

49 Tk . , i 

inis view does not include the portion of ethnographic work to be 

produced by the rural level II contractors, Abt Associates, 
because we have not yet seen it. 

50 o 

Report submitted to the ESP office in the National Institute of Education 
by level II contractor ,4 D. C, Development Associates of San Antonio, 
Texas. 

based on an on-site review of the work of a level II contractor, Scientific 
Analysis Corporation, Berkeley, California. 

52 

,Abt, Wendy. Design Issues in Policy Research: A Controversy . Interim 
Reffort 01- on Abt Associates 1 Pupil Change Study. Cambridge, MA; 
A^t Associates, February 27, 1976. 1 

Welch, Richard. Academic Achievement of Mexican-American Students . 
San Antonio, Texas: D". C. Development Associates, April 1977, 
p. 11^ r 

54 Moore, Vernon, The Third- Report (Vol. 1) (Research Report), Washington, 
D,C.j The National Urban League. Submitted to the office of the 
ESP in the National Institute of Education r ~AprTl 30, 1975. 

/ 
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55 

Interview with staff member of a level II contractor, Abt Associates, 
Cambridge, Massachusetts, March 24, 1977. 

Mdore, Vernon. Student Outcome Report (Vol. 1) (Research Report). Washington, 
D.C.: The National Urban League, Spring 1975, p. 6. 

5 7 

Cervantes, R. , $ Bernard, U. Psychosocial Growth and Academic Achievement 
in Mexican-American Students . San Antonio, Texas: D. C. Develop- 
ment Associates. Submitted to the office of the ESP in the National 
Institute of Education, March 1977. ' 

58 

Internal federal ESP memo from Ray Coward to Norman Gold on the Northwest 
Educational Region;! Laboratory evaluation, dated February 22, 1974. 

/ 

59 

Proposal submitted by Co? tract Research Corporation to the National Institute 
of Education. Level III Evaluation, Documentation and Analysis : 
Technical Proposal . Cambridge, MA. : Contract Research Corporation, 
May 28, 1978, Chapter I, p. 4. 

60 k 

Federal ESP program RFP for the level II evaluation and documentation of 

the ^Experimental Schools Project, undated. 

61 

inters 

District, S^n Antonio, Texas, April J^3^ 

62 * 
Interview with district superintendent of the South Umqua School District, 

South Umqua, Oregon,' May 22 r ,1977, 

63 . ^ # 
Interview with two teachers in Supervisory Union #58, Groveton, New Hampshire, 

June 16, 1977. 

64 

Interview with teacher in the Southeast Alternative School District, 
Minneapolis, Minnesota, May 31, 1977. 

^Interview with teacher in the' E^l^Vpod independent School District, San 
Antonio, Texas, April 12, (19/ 

6 ^The samp phenomenon was experienced by the researchers for this project,, 
despite a concentrated effort to make a distinction between the 
federal government and' the researchers 1 own independent agency. 
Inevitably, the researcher wa$»introduced as being from NIE or Washington. 

67 

Interview with local ESP project director in Supervisory Union #58, Groveton, 
New Hampshire, March 22, 1977. 0 t . 

/ 



Interview with lohal ESP project director in the Edgewood Independent School 
District, Sah Antonio, Texas, April 13, l'977. 
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68 

Interview with local level I ev^Iuator in the Edgewood Independent School 
District, San Antonio, Terfas, April 11 § 13, 1977. 

^Interview with director of a leVel II evaluation, Abt Associates, 
Cambridge, Massachusetts, Augrfs7\6, 1977, 




70 Interview with district superin^dent^fcraig City, Alaska, May 4, 1977. 

^Interview with teacher in the Southeast Alternative School District, 

Minneapolis, Minnesota, May 31, 1977. 
* 

72 

Interview with local ESP project director in the Southeast Alternative 
School District, Minneapolis, Minnesota, March 22, 1977. 

75 Interview with staff member of a level II contractor, Aries Corporation, 
Minneapolis, Minnesota, June 3, 1977. 

74 

I Interview with local ESP project director in, the Edgewood Independent School 
District, San Antonio, Texas, April 13, 1977. 

?5 T . 

Interview with director of a level II evaluation, Human Interaction 

Research Institute, a subcontractor of Scientific Analysis Corporation, 
Berkeley, California, August 31, 1977. 

76, 

Interview with federal ESP staff member, October 28, 1976. 
77 T 

Interview with staff member of a level II contractor, Northwest Educational 

Regional Laboratory, Portland, Oregon, April 14, 1977. # 

* 

78 * * 

Interview with director of a /level II evaluation, Abt Associates, Cambridge, 
s Massachusetts, August 16, 1977. 
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Interview with ethnographer for level II contractor; Abt Associates, 

Cambridge, Massachusetts, April ,19, 1977. Also see Herriot, Robert. 
^ Ethnographic case studies in federally funded multidisciplinary research 
Some design and implementation issues. Anthropology and Education 
Quarterly , May 2, 1977, 8, pp. 106-115. 

80 T 

Interview with ethnographer for level II contractor, Abt Associates, 
Cambridge, Massachusetts, May 24, 1977. 

81 

Interview with director of a level II evaluation, Abt Associates, Cambridge, 
Massachusetts, August 16, 1977. 
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82 

Conversation with federal ESP staff member, October 11, 1977, 
83/ 

w interview with district superintendent of the South Uniqua School District, 
m South Umqua, Oregon, May 23, 1977. 

84 

Interview with local ESP project director in Supervisory Union #58, 
Groveton, New Hampshire, June 9, |977. 

85 * 
Interview with director of a level II evaluation, Northwest Educational 

Regional Laboratory, Portland, Oregon, May 25, 1977. 

86 T 

Interview with director of a level II evaluation, Abt Associates, 
Cambridge^Massachusetts, August 16, 1977. 

87 

Interview with local ESP project director in Supervisory Union #58, 
Groveton, New Hampshire, March 22, 1977. 

88 • . ' 

Interview with local ESP project director in the Franklin Pierce School 

District, Tacoma, Washington, March 30, 1977. 

89* 

Interview with district superintendent of a local rural school site 

(anonyjnity requested), May 2, 1977; interview with district 

superintendent, Craig City, Alaska, May 4, 1977. 

90 ' 

Lpterview with local level I evaluator, Southeast Alternative School District, 
^ Minneapolis, Minnesota, May 31, 1977. 

4 

91 

Interview with local level I evaluator, Piedmont School Project, Greenville, 
South Carolina, May 18, 1977. 

92 

Interview with assistant district superintendent and director of local 

level I evaluation in the Franklin Pierce School District, Tacoma, 
Washington, March 30, 1977. 

93 T ! 

Interview with director of local level I evaluation in the Edgewood 

Independent School District, San Antonio, Texas, April 11, 1977. 

94 

Interview with district superintendent in. the South Umqua School District, 
- South Umqua, Oregon, May 23, 1977. 

95 *f 

After the failure of both the level I and level II evaluations, the 

levei III evaluation never was set under way. It was initially 
designed to rely on the social science findings of the level I 
and level II Revaluations *as a base. for providing feedback to 
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federal and local managers. But "because of the poor quality of 
social science findings from the level I and level II evaluations, 
a level III evaluation hardly seemed possible. Its implementation 
also promised more of the same as found in level I and level II: 
unused social science findings and a new layer of conflict and 
management problems # at both the federal and local levels. For 
example, one local superintendent, when he heard about a level 
III evalQation, expressed what seemed to be a general local feeling: 
he wrote to ESP/Washington and suggested that there be a level IV 
evaluation, because he felt M the hell with them evaluatiag us all 
the time; I decided we should evaluate them for a change." (Inter- 
view with district superintendent of the Carbon County School 
District, Saratoga, Wyoming, April 21, 1977.) 

From the level II contractors 1 perspective, a level III study 
seemed to invite more tensions in their already fragile relation- 
ship with federal and local managers, Consequently, when the 
federal staff raised the possibility of a level III evaluation, 
they received M a lot of flack" from level II contractors. (Inter- 
view with director of a level II evaluation, Abt Associates, 
Cambridge, Massachusetts, August 16, 1977.) 

The general federal feeling toward a level III evaluation was 
expressed by one federal monitor: "Why throw good money after 
bad?" (Interview with former federal ESP evaluation director, 
October 29, 1977.) Given federal opposition to the level III 
evaluation, at least as initially intended, it was finally 
dropped w i t h the departure of its one prominent supporter, 
the original program director. (Interview with federal ESP 
staff member, October 24, 1976.) This decision came as a 
welcome relief to level II contractors and to those local sites 
who had been aware of its possible implementation. (Interview 
with director of a level II evaluation, Abt Associates ^Cambridge, 
Massachusetts, August 16, 1977; interview with district super- 0 
intendent of the Carbon County School District, Saratoga, Wyoming, 
April 27, 1977.) 



V. CONCLUSION 

The Experimental Schools Program was in many important respects 
similar to the Great Society school reform programs it sought to im- 
prove upon. It aimed to solve the problems of children from poor or 
minority-group background^. It sought to do this by extensively re- 
forming local practice. The instruments for reform were federal 
money, ideas, and direction. V In other respects, ESP differed from 
the Great Society "school programs. It was small, for one thing. 
For another, it was able to spend more per pupil than most earlier 
programs.. And finally, its managers were free from many of the polit- 
ical and administrative constraints familiar in other programs, because 
its funds .were discretionary. hSP monies were tied to no legislation 
that spe.cifi^ed how much money was to be spent, or on whom, or for what 
purposes. Thes^ differences should have made it easier for the program 
to achieve its ambitious goals. Thfey had been deliberate, after all, 
part of an effort to design a program' that would overcome the drawbacks 
of earlier programs. Yet by our account, and in the view of many of 
the federal managers, the program failed. It attained none of its 
goals, including its intention to improve on the record of those earlier 
federal programs that ESP managers regularly criticized. Why? And whtft 
lessons may be, learned? ? ^ 

If ESP shared the fate of its predecessors^^ was in part because 
the program- Tasted on naively ambitious notions about school reform. . - 
ESP aimed, for example, at "comprehensive change ,y "ignoring the prac- 
tical wisdom of administration is well as a good deal/ of social sconce, 
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both of which suggest that change in human organizations characterise 
tically occurs ift small, often unrelated ways. These many incremental 
steps sometimes add up to large changes, but they consist of many 
modest unbonnected steps. They are almost impossible to predict, or 
to plan . The hSP program managers had an extra- 

ordinarily overdeveloped faith in local school managers' ability to 
^Revise and implement a grand plan for integrated change. In this they 
resembled their brethren in the Great Society programs, who seamed 

to believe that ideas for reform are better when they are 

larger and more systematic. Such ideas are perhaps for that reason 
more appealing to those wh6se intellectual formation occurred in the 
confines of formal educatidn, where the scholastic values of order and 
tidiness prevail, and where teachers value above all the ability to 
make synthetic formulations. But managing a complex organization is 
pot analogous to writing a good term paper. The skills typically 
rewarded in the latter are often an impediment * to the former. 

It might also have been this curiously scholastic turn of mind 0 
that encouraged *fed£(§l program managers to misunderstand the character 
of local school organizations. They behaved as though local educational 

authorities were organized in a centralized and hierarchical fashion/^ 
as they |^igh^^ pictured in a study of the schoqlsj ^e^m^^ 
Federal managers made^ demands on school superintend 

^deiits consistent with such a tight picture of local organization/ But 
local school' systems are neither tighter tidy/ The consist of indi- . - - 
Vjidual schools that are at least semi-autcmoJimt^ ^itlv^Tespect to the 
central adjninistratidnr-and^^f'teachers within each school who are at least 



semi-autonomous with respect to that school's administration. Each 
local practitioner is free to go his own way, in good part. Any 
change that begins at the center in such a " setting, 

even if it be modest, is, likely to be interpreted in many differ- 
ent ways at the periphery. By pressing local superintendents to devise 
and carry out comprehensive change, federal managers were trying. to pro- 
duce a sort of orderly behavior quite unfamiliar to local organizations. 
For even if some LEAs could plan comprehensively, they would rarely be ablej 
to act in a* coordinated way to implement a broad and integrated plan 
for change. Such things do happen occasionally, but only when the 
pressures for action are extraordinary. ESP, however, was not the 
focu5 of such extraordinary pressures.. Ft was only a program of dis- 
cretionary aid, based on freely contracted arrangements between federal 
and local officials. By behaving as though local school districts 
actually worked as an organization chart might, suggest*, federal offi- 
cials encouraged more resistance to their plans than might have been 

/ generated by other approaches. , ' # 

This synthetic, almost academic turn chHnind was also eyident in 
the ESP program managers' efforts to reform the information that local 
and federal officials used in making decisions. These managers believed^ 
that if they devisj&d^Tliew and more comprehensive scheme for social 
s^ci^nGe''evaliiation, reliable sci^ntiflxHiiforination could replace the 

" ^ knowledge commonly used in managing federal programs and local schools. 
They ignored the great frailties of social science, and the strengths * 
of both ordinary knowledge and the accumulated professional knowledge 
of teachers and administrators. Like many other federal reformers pf 

' • . ' -83 ' . 
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J% • * . 

the last twp decades, *ESP .program managers assumed that social 

I 

science* can ' -ptov^g authoritative guides to action, and 

can strongly influence wh^J teachers atnd administrators do<. In fact, * ? 

•social science is rarely authoritative in. that sense, and particular 

I 

Studies usually influence particular decisions only weakly. The ESP 
evaluation was costly and elaborate, but not irrelevant. Wfyile it 
produced information that was mostly of pool] quality, and not helpful 
to teachers* and administrators; it did add another focus for federal - 

local conflict. . j ' * 

One lesson from this costly affair is that federal^fompetence in 
education, 4 * whatever else it- may Be, does not rest on technically superior 
knowledge. Federal administrators are far removed from local practice, 
both by geographical distance an^by ^e distances introduced by differ- 
ences in the s work of federaLand local managers.' Applied social research 
does nothing to lessen these distances, and in the case of 'the ESP, re- 

4 

search increased the*distarice.*4frederal managers were enthusiastic about 
applied research in part because they had big ideas about loca*l reform, 
but little fiscal and political leverage on Idealities. The presumed 
authority of social science was no sub^tiVute fox* political authority, 

* m * 

^though, at least in this case. /And. the fruits of the research were not 

such as to change any local minds aBout how education , should be carried on 

* . " . * * 

As an instrument Of persuasion, social research was a flop in the ESP. 

,Anot3ier lesson* is th^ federal competence in education doefc not 

extend*/ ,even in easy cases like JSP, to detai/led guidance to l<Qjfaities. 

• "' • " 4 * ' ' • \ 4 < : ' ' < ^ . ^ 

1]he program had no legislative constraints, aftd \t engaged only localities 

* *" * * 

that wighed to 'participate* Vet.sueh was the distance between Washington 
•* V I, * * ♦ 
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and local schools that the efforts of national staff , to offer any more 

... | 

than general guidance about reform were typically fruitless, and often 
problematic. Admittedly, one cannot, generalize freely fpam the case of 
ESP to other, programs — the ESP reform ideas were extraordinarily ambi- 
tious., even in the lexicon of federal education policy. But it does* 
not seem precipitous to argue that if detai led guidance doesn ! t work in 

such a modest, unencumbered, and well -funded endeavor as ESP, it is un- 

* ♦* 

likely to woTk in ^larger, less well-staffed and -funded, and much more 
overburdened programs . 

-•- 

This is not to deny that social Science and federal reform* ideas ■ 
have something to offer education. 'In the case of social science, how- 

k 

ever, federal official* and applied researchers might usefully compare 

♦ ** 
the character of knowledge in the social sciences and the character of 

school practitioners* knowledge and' skil Is , . in order to better* under- 

stand connections and differences between the two realms. Federal . 

research appropriation? are now being cut with the same abandon that 

• * 
earlier marked their application, but in the micist of this the Secretary 

of Education asserted his belief that th.e dissemination of existing social 

science knowledge in v cMtteSfCion would greatly improve' practice. There is 

no doubt that research has something to teach practice ,• anil that researcher: 

can learn much of value from practitioners. But in flight of the record 

» « 

thus far, some closer scrutiny of. hpw this learning .might, occur , and 
how it can usefully be encouragga^feems long^pverdue. ' „ 

Jn the case of federal" reform, matters are. more complex. Some reform 
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efforts, like Title I of the 1965 ESEA, oi» the enforcement of Title VI 
oi the 1964 Civil {lights Act; represent major mpral and political 
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commitments, and the forces at work in such programs are in many respects 
quite different from thosre considered rA this essay. Here we have inves- 
tigated a different species of federal reform--namely , programs which 

express no important national priorities, and for which the chief inspi- 

i * 

ration is not popular pressure, but administrators' or legislators', 

bright ideas about hoK to improve education. How ipight such efforts be 

best managed? One possibility—one of several, liste'd here only by way 

Jfyf ^rample^-is to conceive the federal jole in such local reform as ana- 
* j » - * 

1 ° * 

logous to that of a foundation. The agency gives money to promising „ 

programs on the biasis of the quality of local proposals. Some self-* 

^evaluation and financial accounting are required at the end. And tKe 

grantor hopes that things wdll turn out well in a respectable proportion 

of cases. Foundations have neither the resources nor the authority to 

* 

require more, and in most cases, neither do federal' education agencies. 
Such a' posture would leave,, little room for the overblown 



aspirations of many federal officials, it would reduce ad- 
s' 

♦ ministrative clutter, apd would also improve federal-local relations. # 
Jt might even encourage a more 'accurate picture of the federal role^in 
, certain sorts of local* school reform. 



